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ABSTRACT 


Many  U.S.  Navy  systems  were  built  on  the  fly  and  have  eneountered 
interoperability  problems  at  sea,  sueh  as  erroneous  dual/multiple  traek  designations, 
misidentifieation/traek  identity  eonfliets,  report  responsibility  eonfliets,  friendly  traeks 
displayed  as  unknown/pending,  traeks  dropped  without  operator  aetion,  different  traek 
identities  of  at  different  ships,  ete.  To  identify  and  fix  these  interoperability  problems, 
the  Navy  instituted  the  Distributed  Engineering  Plant  (DEP)  testing  program,  run  by  the 
Naval  Sea  Systems  Command  (NAVSEA),  and  the  End-to-End  (E2E)  testing  initiative, 
eurrently  formed  by  the  Spaee  and  Naval  Warfare  Systems  Command  (SPAWAR). 
Whereas  the  DEP  involves  many  land-based  laboratories  across  the  U.S.  connected  via  an 
Asynchronous  Transfer  Mode  (ATM)  network,  E2E  testing  is  carried  out  entirely  at  one 
laboratory — the  E2E  lab.  The  DEP  testing  program  is  faced  with  the  problem  of 
determining  a  cost-effective  way  of  paying  for  testing — providing  the  participant  DEP 
laboratories  full-time  funding  or  paying  them  on  a  per-test  basis.  A  challenge  faced  by 
the  E2E  testing  program  is  getting  the  E2E  lab  ready  for  testing.  Two  factors 
contributing  to  this  challenge  are  uncertain  availability  of  funding  for  building  the  E2E 
lab  and  the  lack  of  a  comprehensive  plan  to  establish  the  E2E  lab.  Such  a  plan  calls  for  a 
rigorous  justification  of  the  E2E  lab  needs  and  hence  funding  requirements.  This  thesis 
performs  an  in-depth  examination  and  a  qualitative  analysis  of  the  two  testing  programs 
and  a  quantitative  comparative  analysis  of  the  DEP  testing  program’s  paying  options  and, 
using  goal  programming,  provides  data  in  support  of  creating  an  E2E  lab  plan.  The 
significance  of  this  thesis  is  the  use  of  analysis  and  mathematical  programming  to 
provide  analytical  data  in  supporting  informed  decision  making  in  testing  and  evaluation 
of  systems  and/or  systems  of  systems. 
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EXECUTIVE  SUMMARY 


Distributed  and  End-to-End  Testing  in  the  U,S,  Navy 

Dealing  with  test  and  evaluation  of  eombat  and  Command,  Control, 
Communieations,  Computers  and  Intelligenee  (C4I)  systems  in  the  U.S.  Navy,  this  thesis 
foeuses  speeifieally  on  two  separate  Navy  testing  programs  managed  by  two  different 
eommands:  the  Distributed  Engineering  Plant  (DEP)  testing  program,  run  by  the  Naval 
Sea  Systems  Command  (NAVSEA),  and  the  End-to-End  (E2E)  testing  program, 
eurrently  formed  by  the  Spaee  and  Naval  Warfare  Systems  Command  (SPAWAR). 

During  past  deployments  and  Battle  Group  exereises.  Battle  Group  systems 
encountered  interoperability  problems  such  as  erroneous  dual/multiple  track  designations, 
misidentification/track  identity  conflicts,  report  responsibility  conflicts,  friendly  tracks 
displayed  as  unknown/pending,  tracks  dropped  without  operator  action,  different  track 
identities  of  different  ships,  etc.  As  new  technologies  and  commercial-off-the-shelf 
(COTS)  products,  used  in  the  development  of  the  systems,  changed  rapidly  (items  usually 
becoming  obsolete  in  a  couple  of  years),  and  lacked  backward  compatibility,  the  systems 
failed  to  work  together,  thereby  resulting  in  those  interoperability  problems.  When  these 
interoperability  problems  occurred  at  sea,  fixing  them  was  costly  because  of  the  high  cost 
incurred  in  bringing  subject  matter  experts  from  land  to  ships  to  correct  the  problems.  In 
an  effort  to  prevent  these  costly  problems,  the  DEP  testing  program  was  formed  in  June 
1998  to  support  the  evaluation  of  the  interoperability  of  Battle  Group  systems  and  also  to 
aid  in  design  decision  making  early  in  the  systems  acquisition  process. 

The  DEP  consists  of  shore-based  combat  system  sites  such  as  the  Naval  Surface 
Warfare  Centers  (NSWC)  in  Dahlgren,  Dam  Neck,  Wallops  Island,  and  San  Diego;  the 
SPAWAR  Systems  Center-Pacific  (SSC-PAC);  and  the  Naval  Air  Systems  Command 
(NAVAIR)  Paxtuxent/China  Eake.  These  combat  system  sites,  connected  via  an 
Asynchronous  Transfer  Mode  (ATM)  network  provided  by  Defense  Information  Systems 
Agency  (DISA)  and  using  Defense  Information  Systems  Network-Leading  Edge  Services 
(DISN-LES),  replicate,  to  the  maximum  extent  possible,  hardware,  computer  programs, 
connectivity,  and  the  environment  of  the  ship  and  aircraft  combat  systems.  To  replicate  a 

xvii 


Battle  Group,  the  DEP  would  inelude  the  eombat  system  sites  representing  the  members 
of  the  Battle  Group.  A  combat  system  would  be  connected,  not  to  all  sites,  but  only  to 
the  sites  that  could  serve  as  test  beds  for  the  required  platform  combat  systems. 

The  purpose  of  the  E2E  testing  is  to  evaluate  integrated  capabilities  of  shipboard 
C4I  systems  for  interoperability.  The  shipboard  C4I  systems,  which  provide  enhanced, 
integrated  C4I  capabilities  through  integrated  communication  and  information  technology 
systems  that  would  deliver  end-to-end  connectivity  to  aid  in  achieving  decision 
superiority,  need  be  tested  before  their  delivery  to  the  warships,  with  the  hope  to  prevent 
interoperability  failures  from  occurring  while  the  warships  are  at  sea.  In  addition,  not 
only  does  the  E2E  testing  support  the  certification,  but  it  also  supports  the  developmental 
testing  of  multiple  programs.  Concentrated  mainly  on  C4I  systems,  the  E2E  testing 
program  emphasizes  the  area  of  services,  such  as  Domain  Name  Server/Email/Web; 
communication  systems,  such  as  Consolidated  Afloat  Networks  and  Enterprise 
Services/Advanced  Digital  Network  System/Teleport/Network  Operation  Center; 
Information  Assurance,  such  as  Gold  DislCVirtual  Private  Network/Cryptos;  and 
networks,  such  as  Satellite  Communications  &  Eine  of  Sight  &  Pierside.  In  the  future, 
the  E2E  testing  may  be  expanded  to  include  systems  outside  of  the  C4I  arena. 

In  the  E2E  testing  concept,  formulated  in  2007,  a  single  E2E  systems  engineering 
laboratory,  which  is  yet  to  be  built,  replicates  and  tests  these  C4I  systems.  The  E2E  lab  is 
to  house  as  much  C4I  hardware  as  possible  in  order  to  support  the  E2E  testing.  Eimited 
in  funding,  the  E2E  testing  program  plans  to  employ  test  engineers  from  the  Program 
Manager  Warfare  (PMW)  organizations  and  to  have  a  lab  manager,  a  lab  technician, 
network  engineers,  and  systems  administrators  to  perform  day-to-day  lab  activities. 

A  challenge  faced  by  the  E2E  testing  program  is  building  and  getting  the  E2E  lab 
ready  for  testing.  Two  factors  contributing  to  this  challenge  are  uncertain  availability  of 
funding  for  building  the  E2E  lab  and  the  lack  of  a  comprehensive  plan  to  establish  the 
E2E  lab.  Such  a  plan  calls  for  a  rigorous  justification  of  the  E2E  lab  needs  and  hence 
funding  requirements.  Analytical  support  is  needed  to  provide  rigorously  obtained  data 
to  aid  in  the  formulation  of  such  a  plan. 


With  a  focus  on  the  issue  of  eost-effeetive  implementation  of  the  DEP  and  E2E 
testing  programs,  the  researeh  performed  in  this  thesis  involves  (1)  eonducting  a  review 
of  the  past  and  eurrent  test  results  of  the  distributed,  E2E  programs  and  their  arehived 
doeuments,  related  distributed  and  E2E  testing  studies,  and  other  pertinent  related 
material  such  as  test  reports,  test  plans,  and  test  proeedures;  (2)  developing  interview 
questionnaires  direeted  at  managers  of  the  DEP  and  E2E  testing  programs;  (3)  eonducting 
interviews  with  the  DEP  and  E2E  testing  program  managers,  test  direetors,  funetional 
leads,  and  proeess  literature  and  interview  results;  and  (4)  performing  qualitative  and 
quantitative  analysis  and  goal  programming  to  aid  in  determining  the  optimal  eosts  of 
earrying  out  these  testing  methods. 

The  findings  from  this  researeh  follow. 

DEP  Testing  Program 

•  The  DEP  program  currently  pays  the  DEP  partieipant  sites  on  a  per-test  basis, 
rather  than  on  a  retaining  full-time  basis.  Pereeived  by  the  DEP  program  as  a 
eost-effeetive  paying  eoneept,  the  pay-per-test  method  does  not  pay  the  DEP 
sites  to  retain  engineers  upon  the  eompletion  of  testing.  When  a  test  is 
eompleted,  the  DEP  partieipant  sites,  whose  funding  is  now  depleted,  lose  the 
experieneed  engineers.  When  a  new  test  or  retest  is  needed,  newly  hired 
engineers,  who  have  little  or  no  knowledge  of  the  program,  use  a  major 
portion  of  the  alloeated  testing  time  to  learn  and  set  up  the  test.  Also,  when 
any  of  the  DEP  sites  has  trouble  and  eonsequently  eannot  join  the  remaining 
DEP  sites  for  testing  as  planned,  the  latter  have  to  wait  until  the  troubled  site 
is  fixed.  As  a  result,  the  alloeated  testing  time  is  not  fully  used  for  testing. 
The  data  analysis  in  this  research  indicates  that  roughly  38%  of  the  alloeated 
testing  time  is  lost  at  the  beginning  of  a  test.  The  time  loss,  resulting  from  the 
eurrent  pay-per-test  praetice,  remains  a  major  issue  the  DEP  program  needs  to 
resolve. 

•  Another  paying  option  for  the  DEP  program  to  eonsider  is  paying  the  DEP 
partieipant  sites  a  fixed  amount  of  money  to  retain  full-time  engineers.  This 


option  is  not  necessarily  cost-effective  as  the  sites  will  constantly  deplete 
funding,  whether  they  are  doing  testing  or  staying  idle.  Using  the  testing  cost 
data  from  FY01-FY09,  during  which  the  number  of  tests  carried  out  ranges 
from  three  to  eight  per  year,  the  comparative  analysis  performed  in  this 
research,  evaluating  the  two  options — pay-full-time  or  pay-per-test — indicates 
that  conducting  eight  tests  during  FY09  with  two  engineers,  using  the  pay-per- 
test  method  costs  approximately  $3.8M  whereas  using  the  full-time  payment 
method  costs  about  $2.7M.  Furthermore,  the  analysis  indicates  that  if  the 
number  of  tests  per  year  is  not  greater  than  six,  the  pay-per-test  method  is 
cheaper  than  the  pay-full-time  option  and  that  if  more  than  six  tests  per  year 
are  carried  out,  the  full-time  option  is  cheaper  than  the  pay-per-test  option.  As 
six  or  less  tests  are  conducted  for  only  two  out  of  the  nine  years  (FYOl  and 
FY05),  the  pay-full-time  option  would  have  collectively  saved  money  during 
FY01-FY09.  In  this  case,  the  core  experienced  engineers  would  have  been 
likely  retained  and  the  lost  time  would  have  been  reduced,  if  not  eliminated. 
Thus,  in  the  long  run,  if  more  than  six  tests  are  conducted,  savings  will  be 
made  with  the  pay-full-time  option.  Finally,  if  the  number  of  tests  is  less  than 
six  per  year,  the  DEP  program  has  the  flexibility  in  selecting  either  of  the 
paying  options. 

E2E  Testing  Program 

•  Using  testers  from  the  PMWs,  as  currently  planned  by  the  E2E  testing,  is  not  a 
desirable  approach,  as  the  PMW  testers  would  leave  after  completion  of  the 
testing  and  would  also  take  with  them  their  knowledge,  which  might  not  be 
available  when  needed  for  additional  testing.  This  approach  would  possibly 
lead  to  the  time  loss  problem  faced  by  the  DEP  program;  a  lack  of 
accountability  as  different  PMWs  from  which  testers  come  would  claim 
effectiveness  when,  in  actuality,  history  has  shown  otherwise;  linger  pointing 
in  the  case  of  failure;  difficulty  in  timely  coordination  and  flexibility  of 
schedule  in  order  to  support  the  E2E  testing  program;  and  possibly  not 
meeting  the  schedule. 
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•  Rather  being  Program  Objective  Memorandum  (POM)  funded,  the  E2E 
testing  program  secures  its  funding  from  taxing  the  PMWs.  As  such  funding 
is  uncertain,  the  E2E  testing  program  might  not  be  sustainable  in  the  future. 

•  The  first  planned  E2E  testing  event  scheduled  for  December  2008  of  the 
Eincoln  Battle  group  did  not  happen  because  the  E2E  lab  has  not  been  built 
yet.  Eunding  uncertainty  and  the  lack  of  a  complete  plan  to  establish  the  E2E 
lab  contribute  to  the  delay  in  building  and  getting  the  E2E  lab  ready  for 
testing. 

•  Toward  the  formulation  of  a  comprehensive  plan  to  build  the  E2E  lab,  the  use 
of  goal  programming  is  demonstrated  in  this  thesis.  The  demonstration 
reveals  that,  for  the  scenario  treated  in  this  research,  the  planned  cost  for 
building  the  E2E  lab,  which  is  $650,000,  exceeds  the  optimal  cost  obtained 
with  the  goal  programming  approach,  which  is  $647,000,  by  $3,000.  Whereas 
money  savings  are  realized,  the  targeted  number  of  desks  and  chairs  is  short 
by  one.  This  negligible  shortage  is  acceptable. 

•  Einally,  goal  programming  can  be  used  as  a  rigorous  approach  to  determining 
the  goals  of  the  E2E  lab  in  a  timely  fashion  (hence,  saving  money)  that  can 
meet  budgetary  constraints.  The  E2E  testing  program  can  use  this  approach  to 
justify  the  funding  for  the  current  year  and  future  funding.  The  goal 
programming  approach  can  also  be  used  in  support  of  planning  for  programs 
other  than  testing  as  well  as  for  many  multi-goal  problems  encountered  in 
systems  engineering.  Results  obtained  with  this  goal  programming  approach 
are  used  in  decision  making,  in  assessing  the  feasibility  of  achieving  the  goals, 
and  also  in  knowing  how  much  determining  funding  is  required  to  meet  the 
goals. 

Recommendations 

•  Eund  the  DEP  sites  on  a  pay-per-test  basis  if  no  more  than  six  tests  are 
performed  in  a  year;  otherwise,  fund  the  DEP  sites  on  a  full-time  basis. 

•  Retain  the  core  engineers  to  run  DEP  tests. 
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•  Employ  optimization  techniques  in  general  and  goal  programming  in 
particular  in  planning  for  and,  specifically,  in  formulating  a  plan  for  building 
the  E2E  lab. 

•  Employ  rigorous  approaches  —  analytical  and/or  optimization  techniques  — 
from  the  beginning  of  testing  programs  to  plan  for  and  implement  the  testing 
programs. 

Areas  for  Further  Research 

•  Test  and  evaluation  communities  may  extend  the  analysis  and  optimization 
techniques  discussed  in  this  thesis  to  determine  the  optimal  methods  of 
conducting  tests  with  constraints  on  resources  such  as  laboratories,  personnel, 
schedules,  etc. 

•  The  approach  espoused  in  this  thesis  may  serve  as  a  foundation  for  additional 
research  and  studies  of  the  joint  distributed  testing/coalition  distributed 
testing. 
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I.  INTRODUCTION 


A.  BACKGROUND 

This  thesis  deals  with  some  aspeets  of  test  and  evaluation  of  systems  in  the  U.S. 
Navy.  Speeifieally,  this  thesis  foeuses  on  the  distributed  testing  and  end-to-end  (E2E) 
testing  programs.  Briefly,  whereas  distributed  testing  involves  many  geographieally 
distributed  sites  (labs)  that  are  intereonneeted  to  earry  out  a  test,  E2E  testing  is  earried  out 
entirely  in  one  lab  (at  one  site).  By  virtue  of  the  purposes  of  these  test  programs,  only 
eombat  systems  and  eommand,  control,  communications,  computers,  and  intelligence 
(C4I)  systems  are  subjects  of  testing. 

Interoperability  among  the  Elect  units  in  deployment  fails  as  a  result  of  erroneous 
dual/multiple  track  designations,  misidentification/track  identity  conflicts,  report 
responsibility  conflicts,  friendly  tracks  displayed  as  unknown/pending,  tracks  dropped 
without  operator  action,  etc.  (Monteith,  2001).  In  the  absence  of  a  test  and  evaluation 
program,  fixing  these  interoperability  failures  adversely  affected  the  Elect’s  missions.  In 
one  instance,  a  Battle  Group  was  deployed  without  two  modern  combatants,  as  the  latter 
were  tied  to  a  pier  in  order  to  have  interoperability  problems  fixed.  In  another  instance, 
a  great  deal  of  time  during  the  final  six  months  prior  to  a  Battle  Group  deployment  was 
consumed,  not  by  training,  but  by  “debugging”  of  systems  in  the  Battle  Group.  These 
unacceptable  instances  called  for  a  systematic  testing  and  evaluation  approach  to  deal 
with  the  interoperability  problems  (DEP,  2005). 

To  address  combat  systems  interoperability  problems  across  Battle  Management 
Command,  Control,  Communications,  Computers  and  Intelligence  (BMC4I)/combat 
systems  and  to  work  with  the  Elect  to  fix  the  interoperability  problems,  the  Naval  Sea 
Systems  Command  (NAVSEA),  in  April  1998,  formed  the  Combat  System 
Interoperability  Task  Eorce,  whose  objectives  were  to  examine  the  interoperability  crisis 
and  to  provide  recommendations  and/or  solutions  for  fixing  the  interoperability  problems. 
To  achieve  these  objectives,  the  Task  Eorce  determined  the  feasibility  and  cost  of  using  a 
land-based  distributed  engineering  plant  (DEP)  to  support  the  design,  development. 
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testing,  and  evaluation  of  Battle  Group  systems  (DEP,  2005).  In  June  1998,  the  DEP  was 
established,  signifying  the  beginning  of  the  Navy’s  distributed  testing. 

Distributed  testing  has,  however,  eneountered  the  diffieulty  of  getting  all  the  labs 
together  and  having  all  resources  available  at  one  time  to  form  the  DEP.  The  DEP 
program  currently  pays  the  DEP  participant  sites  on  a  per-test  basis,  rather  than  on  a 
retaining  full-time  basis  (Mann,  2004).  Perceived  by  the  DEP  program  as  a  cost-effective 
paying  concept,  the  pay-per-test  method  does  not  pay  the  DEP  sites  to  retain  engineers 
upon  the  completion  of  testing.  When  a  test  is  needed,  inexperienced  part-time  engineers 
are  hired  to  perform  the  test.  This  practice  has  led  to  testing  inefficiency,  namely,  the 
loss  of  test  time  to  setting  up  the  labs  by  inexperienced  part-time  engineers,  and  need  to 
be  evaluated  against  the  full-time  paying  option.  One  of  the  goals  of  this  thesis  research 
is  to  perform  analyses  to  aid  the  DEP  program  manager  (PM)  in  making  the  correct 
decision  in  paying  the  DEP  labs  for  testing.  Chapter  II  discusses  in  detail  the  DEP  testing 
program  and  its  problems/issues. 

The  concept  of  E2E  testing  in  the  Navy  began  in  2007,  when  Office  of  the  Chief 
of  Naval  Operations  (OPNAV)  tasked  the  Navy  Program  Executive  Officer  Command, 
Control,  Communications,  Computers,  and  Intelligence  (PEO  C4I)  to  provide  enhanced, 
integrated  C4I  capabilities  through  integrated  communication  and  information  technology 
systems  that  would  deliver  end-to-end  connectivity  to  aid  in  achieving  decision 
superiority  (Miller,  2008).  These  integrated  communication  and  information  technology 
systems  often  make  use  of  commercial  of  the  shelf  (COTS)  technologies.  When  these 
systems  encounter  problems  at  sea,  fixing  them  can  become  costly  because  of  the  high 
cost  incurred  in  bringing  subject  matter  experts  (SMEs)  from  land  to  ships  to  correct  the 
problems  at  hand.  Requested  by  PEO  C4I  to  provide  support  in  solving  this  cost 
problem,  the  Space  and  Naval  Warfare  (SPAWAR)  Systems  Center-Pacific  (SSC-PAC) 
is  building  an  E2E  Systems  Engineering  (SE)  lab  (E2E  SE,  2008),  which  will  replicate 
and  test  multiple  shipboards  C4I  systems  before  their  delivery  to  the  Elect.  Building  the 
E2E  lab  can  be  costly,  and  the  utilization  of  the  E2E  lab,  along  with  the  use  of  personnel, 
is  also  an  issue.  The  second  goal  of  this  thesis  research  is  to  analyze  the  current  approach 
to  building  the  E2E  lab  and  to  explore  the  use  of  goal  programming  as  an  optimization 
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tool  to  aid  the  E2E  testing  PM  in  optimally  formulating  a  plan  to  build  the  E2E.  Chapter 
III  discusses  in  detail  the  E2E  testing  effort  and  its  problems/issues. 

The  rest  of  this  chapter  discusses  the  purpose  of  the  thesis  (Section  B),  the 
research  questions  (Section  C),  the  potential  benefits  of  the  thesis  (Section  D),  the  scope 
and  the  research  methodology  (Section  E),  and  the  organization  of  the  remaining  of  the 
thesis  (Section  E). 

B.  PURPOSE 

With  a  focus  on  the  issue  of  cost-effective  implementation  of  the  DEP  and  E2E 
testing  programs,  the  research  performed  in  this  thesis  identifies  the  issues  and  problems 
encountered  by  the  DEP  and  E2E  testing  programs;  examines  available  information  and 
data  from  the  two  programs;  performs  qualitative  and  quantitative  analyses  to  provide 
data  to  aid  the  DEP  testing  program  with  its  implementation;  and  explores  and  applies 
goal  programming  to  support  the  formulation  of  a  plan  to  build  the  E2E  lab. 

C.  RESEARCH  QUESTIONS 

The  primary  questions  are: 

1 .  How  does  the  Navy  employ  distributed  testing  and  E2E  testing? 

2.  Can  optimization  methods  be  used  to  aid  in  determining  the  minimal  cost  in 
implementing  these  testing  methods  while  maximizing  the  use  of  available 
resources  (personnel  and  lab  hardware)? 

Answering  these  questions  amounts  to  answering  the  following  secondary 
questions: 

a.  What  are  the  differences  between  the  distributed  testing  and  E2E  testing 
programs? 

b.  What  are  the  current  methods  used  by  the  Navy  to  conduct  distributed  and 
E2E  testing? 

c.  How  does  the  Navy  employ  these  testing  methods  for  testing  of  naval  combat 
systems? 
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d.  What  analysis  and  optimization  methods  that  can  be  used  to  aid  in 
determining  the  minimal  costs  of  carrying  out  these  testing  methods  while 
maximizing  the  use  of  lab  resources  (including  hardware  and  personnel)? 

e.  What  recommendations  regarding  the  implementation  of  these  testing 
methods  can  be  provided  to  the  Navy? 

D,  RESEARCH  BENEFIT 

This  thesis  demonstrates  the  analysis  and  the  use  of  goal  programming  methods 
that  can  be  used  by  PMs  to  effectively  and  efficiently  perform  their  tasks  of  budgeting, 
scheduling  of  lab  assets,  and  utilization  of  the  labs.  In  addition,  the  Test  and  Evaluation 
community  may  extend  the  analysis  and  optimization  techniques  discussed  in  this  thesis 
to  determine  the  optimal  methods  of  conducting  tests  with  constraints  on  resources  such 
as  laboratories,  personnel,  schedules,  etc.  Furthermore,  the  approach  espoused  in  this 
thesis  may  serve  as  a  foundation  for  additional  research  and  studies  of  the  joint 
distributed  testing/coalition  distributed  testing. 

E.  SCOPE  AND  METHODOLOGY 

1.  Scope 

With  a  focus  on  the  issue  of  cost-effective  implementation  of  the  DEP  and  E2E 
testing  programs,  the  research  performed  in  this  thesis  identifies  the  issues  pertaining  to 
the  two  major  problems  encountered  by  the  DEP  and  E2E  testing  programs;  examines 
available  information  and  data  from  the  two  programs;  performs  qualitative  and 
quantitative  analyses  to  provide  data  to  aid  the  DEP  program  with  its  implementation; 
and  explores  and  applies  goal  programming  to  support  the  formulation  of  a  plan  to  build 
the  E2E  lab. 

2.  Methodology 

The  research  methodology  involves: 

a.  Conducting  a  review  of  the  past  and  current  test  results  of  the 
distributed,  E2E  programs  and  their  archived  documents,  related 
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distributed  and  E2E  testing  studies,  and  other  pertinent  related  material 
sueh  as  test  report,  test  plans,  and  test  proeedures; 

b.  Developing  interview  questionnaires  directed  at  managers  of  the  DEP 
and  E2E  testing  programs; 

c.  Conducting  interviews  with  the  DEP  and  E2E  testing  PMs,  test 
directors,  functional  leads,  and  process  literature  and  interview  results; 

d.  Performing  qualitative  and  quantitative  analysis  and  goal 
programming  to  aid  in  determining  the  optimal  costs  of  carrying  out 
these  testing  methods;  and 

e.  Eormulating  recommendations  to  the  Navy  regarding  the 
implementation  of  these  testing  methods. 

F.  THESIS  ORGANIZATION 

The  rest  of  the  thesis  is  organized  as  follows.  Chapter  II  discusses  the  DEP 
testing  program.  Chapter  III  discusses  the  E2E  testing  program.  Chapter  IV  presents  the 
analyses  of  both  the  DEP  and  E2E  testing  programs.  Chapter  V  discusses  the 
comparative  analysis  for  aiding  the  DEP  testing  program  in  making  paying  decisions  and 
the  goal  programming  approach  in  supporting  the  E2E  testing  program.  Chapter  VI 
contains  the  conclusions  and  recommendations  on  future  research. 


5 


THIS  PAGE  INTENTIONALLY  LEET  BLANK 


6 


II.  DISTRIBUTED  ENGINEERING  PLANT  PROGRAM 


A,  INTRODUCTION 

This  chapter  provides  the  background  of  and  the  problems  encountered  by  the 
DEP  program.  As  mentioned  in  Chapter  I,  the  Navy  DEP  was  formed  to  aid  in  solving 
interoperability  problems  sueh  as  eommunications  between  ship  systems,  eommon 
operational  pictures  between  ships,  incorreet  IDs,  dual  tracks,  etc.  that  occurred  during 
deployments  and  Battle  Group  exercises  (Monteith,  2001).  Interoperability  problems 
encountered  by  eombat  ships  have  had  their  roots  in  the  use  of  commereial-off-the-shelf 
(COTS)  products  and  new  technologies  to  develop  systems.  As  technologies  change 
rapidly  (items  usually  becoming  obsolete  in  a  couple  of  years)  and  baekward 
eompatibility  of  some  items  has  been  a  common  phenomenon,  the  use  of  COTS  and 
constant  technological  advances  has  ereated  aequisition  problems  not  only  for  the  Navy, 
but  also  for  the  Department  of  Defense  (DoD).  Interoperability  problems  have  become  a 
challenge  to  the  Navy. 

In  Eebruary  1998,  the  Eleet  reported  concerns  regarding  interoperability  failures 
among  combat  systems  that  were  reeently  installed  in  the  Eleet  units  (USS  John  E. 
Kennedy  and  USS  Wasp).  Debugging  of  glitehes  eonsumed  a  great  deal  of  Eleet  time 
during  the  final  six  months  prior  to  Battle  Group  deployment  (DEP,  2005).  In  March 
1998,  the  Chief  of  Naval  Operations  assigned  the  Naval  Sea  Systems  Command 
(NAVSEA)  the  responsibility  to  address  combat  systems  interoperability  problems  across 
BMC4I/combat  systems  and  to  eoordinate  an  effort  with  the  Eleet  to  solve  the 
interoperability  problems.  In  April  1998,  NAVSEA  formed  the  Task  Eorce  on  Combat 
System  Interoperability  to  study  the  interoperability  crisis  and  to  provide 
recommendations  for  solutions.  In  June  1998,  the  Task  Eoree  affirmed  the  teehnieal 
feasibility  of  the  establishment  of  a  DEP  to  support  the  evaluation  of  the  interoperability 
of  battle  force  systems  and  to  enable  good  design  decisions  early  in  the  aequisition 
process  (DEP,  2005). 
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Following  the  Task  Force  Report,  fourteen  Navy  organizations  shown  in  Figure  1, 
representing  the  surface,  air,  subsurface,  and  C4I  components  across  all  Navy  Systems 
Commands  (SYSCOMs),  formed  the  Battle  Force  Interoperability  Navy  Alliance  (DEP, 
2005)  to  cooperatively  support  the  interoperability  task.  The  initial  purpose  of  the  Navy 
Alliance  was  to  develop  a  proposal  for  the  establishment  and  implementation  of  a  Navy 
DEP. 


^feva  I  Sur  fi  ce  Warfare  Gen  far/ CO  -Cfah  ^ren,  VA 
Aeg  8  Com  ba  t  S  ysfa  ns  C  ent  er  -  Wal  ops  I  si  and,  V  A 
^fava  I  Warfare  Mai  ysB  Sta  ton  -Co  o  na,  CA 
^fava  I  Lhde  s  ea  Warfare  Cfanter  -Na  vsp  ort  R 
^fava  I  Sur  fa  ce  Warfare  Cfan  far/ Pl-D  -Oxnard,  CA 
SPA  WV  R  Sys  fa  ns  C  ent  er-  P  acf  I  c-  S  an  Cfa  go,  C  A 

f«fava  I  Sur  fa  ce  Warfare  Gen  far/  PHD  -D  am  N  eck,  V  A 

SPA  W^  R  Sys  fa  ns  C  ent  er-  A I  ant  c  -  C  har  fa  sb  n,  S  C 

Hbva  I  Sur  fa  ce  Warfare  Gfan  far/  PHD  -S  an  Dfa  go,  CA 

Aeg  8  Tran  hg  and  F^a  dh  ess  Gfan  far  -  D  ahl  gren,  V  A 
l^va  I  F^se  arcKi  Laborat  ory  -A'Ihgton  ,VA 
JHLI  A  ppl  fad  P  Fiysl  cs  Labo  rab  y  -  Laurel ,  Ik/I  D 
l^va  I  A  rWarfareC  enter//^  -Patuxe  ntRirer,  l\/D 
l^va  I  A  rWarfareC  enterWD  -Ghi  na  L  ake,  C  A 


Eigure  1 .  NAVY  DEP  Alliance  (DEP,  2005) 


The  DEP  consists  of  shore-based  combat  system  sites  such  as  the  Naval  Surface 
Warfare  Centers  (NSWC)  in  Dahlgren,  Dam  Neck,  Wallops  Island,  and  San  Diego;  the 
SPAWAR  Systems  Center-Pacific  (SSC-PAC);  and  the  NAVAIR  Paxtuxent/China  Lake. 
These  combat  system  sites  replicate,  to  the  maximum  extent  possible,  hardware, 
computer  programs,  connectivity,  and  the  environment  of  the  ship  and  aircraft  combat 
systems  (DEP,  2005).  To  replicate  a  Battle  Group,  the  DEP  is  extended  to  include  the 
combat  system  sites  representing  the  members  of  the  Battle  Group.  A  combat  system 
consists  of  many  tightly  integrated  key  elements.  It  is  the  brain  of  the  ship,  which 
acquires  track  data,  intelligent  data,  situational  awareness  data,  and  displays  these  data  on 
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its  Display  console.  It  interfaces  with  other  subsystems  sueh  as  radar,  target  aequisition, 
Cooperative  Engagement  Capability  (CEC),  and  the  Global  Command  and  Control 
System  Maritime  (GCCS-M),  ete  (DEP,  2005).  A  combat  system  is  to  be  eonneeted,  not 
to  all  sites,  but  to  only  the  sites  that  serve  as  test  beds  for  the  required  platform  eombat 
systems.  The  DEP  is  thus  supposed  to  be  the  most  eost-effective  way  to  perform 
interoperability  testing. 

The  remaining  of  this  ehapter  is  organized  as  follows.  Seetion  B  diseusses  the 
Navy  systems  tested  with  the  DEP;  Seetion  C  the  evolution  of  DEP  testing  objeetives  and 
names;  Seetion  D  the  DEP  network  arehiteeture;  Seetion  E  DEP  test  seheduling;  Seetion 
E  DEP  staffing;  and  Seetion  G  DEP  test  results.  Seetion  H  provides  a  brief  summary  of 
the  ehapter. 

B,  NAVY  SYSTEMS  TESTED  WITH  THE  DEP 

The  DEP  is  intended  to  test  all  of  the  Navy’s  systems,  speeilieally  eombat 
systems  as  they  are  the  most  eleotronieally  integrated  and  their  performanee  is  related  to 
safety  required  in  firing  weapons.  DEP  testing  would  involve  the  Aegis  Weapons  System 
(AWS),  Advaneed  Combat  Direetion  System  (ACDS),  radar  system,  navigation  system, 
taetieal  data  link  systems,  and  the  Global  Command  and  Control  Maritime  (GCCS-M) 
system.  As  a  battle  group  has  many  ships,  whieh  in  turn  have  many  eombat  systems 
(e.g.,  ACDS,  AWS),  one  single  lab  (site)  would  not  be  able  to  house  all  or  many  systems, 
and  labs  (sites)  aeross  the  eountry  would  be  needed  in  testing  all  of  these  systems  as 
illustrated  in  Table  1  (Coyne,  2006). 

Both  Integrated  Combat  Systems  Test  Detaehment  (ICSTD)  in  San  Diego,  CA 
and  Combat  Direetion  Systems  Aetivity  (CDSA)  in  Dam  Neek,  VA,  house  the  Advaneed 
Combat  Direetion  System  (ACDS)  system;  both  the  Integrated  Weapons  Systems  Eab 
(IWSE)  in  Dahlgren,  VA,  and  the  Surfaee  Combat  Systems  Center  (SCSC)  in  Wallops 
Island,  VA,  house  the  Aegis  Weapons  System  (AWS);  the  SPAWAR  Systems  Center — 
Paeifie  (SSC-PAC)  in  San  Diego,  CA,  houses  the  Global  Command  and  Control  System 
Maritime  (GCCS-M)  systems;  both  the  Patuxent  River,  MD  and  SSC-PAC  in  San  Diego, 
CA,  house  an  E-2  Hawkeye  (E2C)  system;  and  China  Take  houses  P-18.  Systems 
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subjected  to  DEP  testing  include  a  Cooperative  Engagement  Capability  (CEC),  Auto 
Identification  (ID),  Common  Data  Eink  Management  System  (CDLMS)/Command  and 
Control  Processor  (C2P),  Ships  Gridlock  System/Automatic  Correlation  (SGS/AC), 
Target  Acquisition  System  Mark-23  (TAS  MK-23),  Interrogator  Set  (TPX-42),  Integrated 
Automated  Detection  and  Tracking  (lADT)  System  (SYS-2),  Eink-4/I  I/I6,  and  Air 
Defense  System  Integrator  (ADSI).  Weapons  or  sensors  that  are  unavailable  to  a  combat 
system  are  then  emulated  by  computer  simulations  or  a  stimulator  (SIMS/STIMS).  In 
addition,  the  SIMS/STIMS  generates  a  common  environment  representing  the  real  world 
and  entities  therein.  The  emulated  battle  group  then  performs  within  a  controlled, 
repeatable  environment  under  the  close  scrutiny  of  engineers  and  developers  (DEP, 
2005). 


Facility/Lab 

Combat  Systems 

Numbers  of  Systems 

Available  For  Use 

Naval  Surface  Warfare  Center,  Port  Hueneme  Division, 
Detachment  San  Diego,  CA  (NSWCPHD  -  ICSID) 

SSDS  MK  2  Mod  1/2  and  ACDS  ELK 

0/1 

3  (dependent  on 
configuration  requested) 

Naval  Surface  Warfare  Center,  Combat  Direction  Systems  Activity 
Dam  Neck,  VA  (NSWC  CDSA) 

SSDS  MK  2  Mod  1,  ACDS  Block  0/1, 
ADSI,  and  CDS 

3 

Integrated  Warfare  Systems  Laboratory,  Dablgren,  VA.  (IWSL) 

AWS  5/6/7  Baseline 

2 

Center  for  Surface  Combat  Systems,  Dablgren,  VA.  (CSCS) 

AWS  2/5/6  Baseline 

2 

Surface  Combat  Systems  Center,  Wallops  Island,  VA.  (SCSC) 

AWS  2/5/6/7  Baseline  and  SSDS 

2 

Naval  Air  Warfare  Center,  E2C  System  Test  and  Evaluation 
Laboratory,  Patuxent  River,  MD.  (NAWCAD  PAX  -  ESTEL) 

Ha\^eye  2000  with  Mission  Computer 
Unit(MCI]) 

1 

Naval  Air  Warfare  Center,  Avionics  Integration  Laboratory,  China 
Lake,CA.  (NAWCAD -AIL) 

F/A-18  (C,  D,  and  F)  Hornet 

2 

Space  and  Naval  Warfare  Systems  Command,  Systems  Center  San 
Diego,  CA.  Group  LI  E2C  Test  Laboratory.  (SSC-SD  -  E2C) 

Group  nE2C  (multiple  software  builds 
supported) 

1 

Space  and  Naval  Warfare  Systems  Command,  Systems  Center  San 
Diego,  CA.  Reconilgurable  Land-Based  Test  Site  for  Global 
Command  and  Control  System  -  Maritime.  (SSC-SD  -  RLBTS) 

(GCCS-M  elements  only).  GCCS-M 
(multiple  hull  and  system  configurations 
possible). 

3  or  more 

Naval  Surface  Warfare  Center,  Dablgren  Division,  DEP  Operations 
Center,  Dablgren,  VA.  (NSWCDD-DOC) 

Netwnrk/infrastructure,  DIS,  and 
TACCOMM  monitoring^analysis, 
scenario  controFbroadcast  only 

N/A 

Space  and  Naval  Warfare  Systems  Command,  Systems  Center  San 
Diego,  CA.  Systems  Integration  Facibty.  (SSC-SD  -  SIF) 

ADSI,  Data  Link  (Link-4/11/16),  and 
TACCOMM  monitoring^analysis. 

N/A 

Space  and  Naval  Warfare  Systems  Command,  Systems  Center  San 
Diego,  CA.  Network  Operations  Center.  (SSC-SD -NOC) 

Network/infrastructure  operations, 
monitoring,  and  analysis  only 

N/A 

Naval  Surface  Warfare  Center,  Corona,  CA.  (NSWC  Corona) 

Voice  and  TACCOMM  monitoring  and 
analysis 

N/A 

Table  1.  Tabs  with  Associated  Combat  Systems  (Coyne,  2006) 
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C.  EVOLUTION  OF  DEP  TESTING  OBJECTIVES  AND  NAMES 

The  original  plan  of  the  DEP  stopped  with  “Battle  Foree  Interoperability  Test” 
(BEIT),  focused  on  the  area  of  Air  &  Surface  Warfare  (ASW)  and  C4L  To  conduct 
testing,  the  DEP  used  the  Navy  D-30  process  (DEP,  2005)  to  manage  and  test  software 
before  its  delivery  to  the  Fleet.  To  justify  future  funding  to  sustain  the  DEP  program,  the 
DEP  changed  its  testing  objectives  and  the  test  name  from  BEIT  to  Force  Interoperability 
Test  (FIT).  In  addition  to  focusing  on  software  upgrades,  DEP  testing  now  also  focused 
on  inputs  from  the  Battle  Group  commanders  and  sailors.  With  their  inputs,  the 
interoperability  problems  were  reproducible  in  the  lab  environment  and  the  proposed 
workaround  along  with  the  proposed  capabilities  and  limitations  were  then 
communicated  to  the  Battle  Groups.  FIT  was  then  changed  to  Interoperability 
Assessment  (lA)  in  2005,  which  was  then  changed  again  in  2006  to  Basic  Platform 
Interoperability  Assessments  Test  (BPIT)  and  Advanced  Platform  Interoperability 
Assessments  Test  (APIT).  Subsequently,  the  name  was  changed  to  Interoperability 
Development  (I/O  DEV)  and  Interoperability  Certification  (I/O  CERT),  which  have  thus 
far  remained  unchanged  (BFIMS,  2009). 

Whereas  the  DEP  concept  remains  unchanged,  which  is  to  ensure  that 
interoperability  and  software  problems  are  flushed  out  before  delivery  to  the  Fleet,  the 
DEP  testing  requirements  changed  with  the  name  change.  Instead  of  testing  multiple 
Battle  Groups,  the  DEP  testing  has  been  concentrated  on  just  one  Battle  Group  with  an 
emphasis  on  certification  of  new  or  upgraded  software. 

D.  DEP  NETWORK  ARCHITECTURE 

As  discussed  in  Chapter  I,  the  basic  idea  underlying  the  DEP  is  to  connect  the  labs 
(sites)  that  house  the  systems  identified  in  Section  B  of  this  chapter  to  emulate  a  Battle 
Group  with  its  platforms  and  its  operational  environment.  The  inter-site  connectivity 
emulates  the  various  communications  networks  connecting  the  elements  of  a  strike  group 
and  “back  channel”  communications  for  test  coordination  and  data  collection.  Computer 
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simulations  provide  coordinated  stimuli  to  the  various  platform  subsystems  at  each  site. 
The  stimuli  would  be  both  representative  of  the  operational  environment  and  repeatable 
over  successive  test  runs  (DEP,  2005). 

In  some  DEP-unrelated  efforts,  connecting  the  various  laboratories  using  Secret 
Internet  Protocol  Router  Network  (SIPRNET)  has  failed,  because  of  its  low-bandwidth, 
its  lack  of  around-the-clock  availability,  and  attending  laborious  security  paperwork.  In 
the  DEP  approach,  the  laboratories  are  connected  via  an  Asynchronous  Transfer  Mode 
(ATM)  network  using  the  Defense  Information  System  Network  -  heading  Edge  Services 
(DISN-EES).  The  high-speed  ATM  network  is  leased  from  AT&T.  The  DEP  ATM 
network  operates  at  10  Mbps,  but  it  could  also  operate  at  45  Mbps  (at  a  relatively  low 
cost).  Growth  to  155  Mbps  and  beyond  is  achievable.  This  network  is  available  24/7; 
once  the  network  is  credited  (usually  once  a  year),  it  can  be  used  at  anytime  (DEP,  2005). 

E.  DEP  TESTING  SCHEDULING 

The  first  DEP  test  was  performed  on  the  replicated  Eincoln  and  Truman  Battle 
Groups  in  1999,  whose  systems  (marked  with  ‘X’)  are  listed  in  Table  2  (Seaver,  2004a). 
This  first  test  was  successfully  conducted,  despite  the  difficulty  associated  with  the  newly 
formed  DEP  initiative  and  caused  by  the  unavailability  of  the  labs  already  scheduled  for 
some  other  events.  As  the  DEP  program  matured  and  the  fleet  requirements  changed 
(i.e.,  testing  not  only  ASW  but  also  C4I  systems),  the  support  from  some  of  these  labs 
began  to  vanish;  only  a  handful  of  the  labs  are  now  actually  being  utilized.  Even  with  the 
smaller  number  of  labs  involved,  scheduling  DEP  tests  still  complicate  the  schedules  and 
planning  of  the  labs  involved  in  DEP  testing,  as  the  labs  would  have  to  modify  their 
schedules,  which  had  been  established  months  in  advance.  Eurthermore,  if  the  DEP  test 
date  changed,  then  the  DEP  had  to  re-schedule  the  labs  again,  which  would  create  major 
scheduling  conflicts  for  the  labs. 

When  a  few  components  malfunction  during  a  test,  the  test  has  to  be  repeated.  To 
handle  the  scheduling  of  tests  and  retests,  the  DEP  uses  a  web-based  scheduling  tool  to 
schedule  the  labs  (DEP,  2005).  This  tool  effectively  aids  the  PM  and  site  leads  in 
scheduling  and  re-scheduling  the  labs.  The  tool  automatically  sends  an  email  to  the 
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participating  sites  alerting  a  need  for  lab  aeeess  and  an  email  to  the  scheduling  manager 
indicating  the  availability  of  the  requested  lab.  If  a  schedule  conflict  arose,  the 
scheduling  manager  would  then  determine  a  feasible  time  for  all  parties  involved  in 


testing. 


CVN 

LHA 

CG 

DDG 

DD 

FFG 

E-2  GPn 

F-I4 

COMBAT  SYSTEM 

ACDS  Blk  0 

ACDS  Blk  0 

AWS 

AWS 

CDS 

CDS 

J9VETCBA 

D03B 

C2P 

X 

N/A 

X 

X 

N/A 

N/A 

N/A 

N/A 

SGS/AC 

X 

X 

X 

X 

N/A 

N/A 

N/A 

N/A 

Auto  ID 

X 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

TAS  MK-23 

X 

X 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

TPX-42(V)8/13/14 

X 

X 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

SYS-2 

X 

X 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

GCCS-M 

X 

X 

X 

X 

X 

X 

N/A 

N/A 

LINK  A/J/SAT  J 

X 

X 

X 

X 

D 

D 

A/J 

J 

X  -  Indicates  system  under  test 

Table  2.  List  of  DEP  Systems  (Seaver,  2004a) 


F.  DEP  STAFFING 

Staffing  has  proved  to  be  most  challenging  as  it  is  related  to  funding  and  testing 
tools  issues.  Sinee  DEP  testing  is  not  running  on  a  eontinuous  basis,  the  testers  are  paid 
only  when  they  support  the  testing.  In  other  words,  the  testers  are  paid  on  a  part-time 
basis  (i.e.,  per-test  basis).  The  total  funding  alloeated  to  a  site  manager  is  based  on  an 
estimated  number  of  tests  to  run  per  year  (Avarez,  2006).  When  the  number  of  tests 
unexpeetedly  inerease,  the  per-test  based  yearly  funding  alloeation  will  not  be  suffieient 
to  eover  the  testers. 

Many  years  of  DEP  testing  have  shown  that  a  test  has  never  been  earned  out  on 
time.  The  delay  in  testing  is  eaused  partly  by  hardware  and  software  problems  and  partly 
by  the  high  turnover  rate  of  personnel.  Usually,  testers  would  stay  in  their  part-time 
position  and  leave  to  aeeept  an  available  full-time  position  at  another  organization. 
Often,  the  new  personnel,  who  replaee  the  departing  testers,  have  little  or  no  knowledge 
of  the  eondueted  tests  and  therefore  are  unable  to  troubleshoot  problems  that  arise  during 
the  testing.  Although  this  high  turnover  rate  of  personnel  has  become  problematic  to  the 
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DEP  program,  the  DEP  PM  has  not  been  willing  to  pay  full-time  positions  to  retain  these 
SMEs.  The  DEP  issues  of  test  delay,  the  pay-per-test  eoncept,  and  a  paying  alternative 
are  diseussed  in  more  detail  in  Chapter  IV. 

G.  TEST  RESULTS 

Many  DEP  tests  have  discovered  a  few  new  problems  and  repeatedly  revealed 
identical  interoperability  problems  (Mahon,  2003b).  These  interoperability  problems 
thus  have  not  been  fixed.  The  new  problems  are  reported  to  the  program  offices  for 
corrections.  Some  of  the  interoperability  problems  have  already  been  documented  as 
trouble  reports  (TRs)  in  their  local  databases.  If  the  TRs  were  high  priority  which  did  not 
have  the  workaround,  then  corrections  would  be  made  during  the  next  software  upgrade. 
The  TRs  have  also  been  presented  to  the  Battle  Group  personnel. 

Eessons  learned  and  the  workaround  solutions  have  been  very  useful  to  the  Elect 
as  the  sailors,  rather  than  the  engineers  and  testers,  now  could  use  them  to  figure  out  the 
causes  of  interoperability  failures  and  to  do  the  fixing  at  sea.  The  lessons  learned  have 
been  also  shared  with  the  system  developers,  but,  parenthetically,  in  the  absence  of  any 
mandating  instructions  to  consider  them,  the  developers  largely  tend  to  ignore  the  lessons 
learned.  The  DEP  program  has  never  implemented  the  lessons  learned  in  the 
development  of  systems,  because,  except  for  a  few  engineers  called  in  occasionally  for 
root  cause  analysis,  the  engineers  doing  interoperability  testing  are  not  the  engineers  who 
developed  the  subsystems  and  also  because  the  system  developer  would  not  be  around  to 
have  tests  repeated  to  deal  with  problems  discovered  during  post-test  data  analysis  and  to 
subsequently  incorporated  the  problem  resolution  in  the  design  of  the  subsystems. 

H.  SUMMARY 

To  summarize,  the  DEP  concept  is  to  alleviate  of  the  cost  of  trying  to  reproduce 
the  interoperability  problems  encountered  during  the  Elect  operations  because  of  the  huge 
cost.  To  be  able  to  reproduce  the  interoperability  problems  in  the  DEP  lab  environment 
is  an  incredible  accomplishment,  but,  after  several  tests,  the  same  problems  such  as 
duplicated  track  numbers,  dual  tracks,  etc.  have  been  still  observed.  In  addition,  the 
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turnover  of  the  testing  personnel  has  been  also  a  dilemma.  Being  unable  to  retain  the 
eore  SME  to  support  testing  and  to  fix  interoperability  problems,  the  DEP  might  beeome 
one  of  the  programs  that  have  no  value  added  to  the  war  fighter.  The  DEP  should  be 
analyzed  objeetively  for  its  effectiveness. 


15 


THIS  PAGE  INTENTIONALLY  LEET  BLANK 


16 


III.  END-TO-END  TESTING  PROGRAM 


A,  INTRODUCTION 

This  chapter  provides  the  background  of  and  the  issues  encountered  in  the  E2E 
program.  The  Navy  E2E  testing  program  began  in  2007,  when  the  Navy  PEO  C4I  was 
tasked  to  provide  enhaneed,  integrated  C4I  eapabilities  through  integrated 
eommunieations  and  information  teehnology  systems  to  aid  in  aehieving  battle  deeision 
superiority  (Miller,  2008).  These  integrated  eommunieations  and  information  teehnology 
systems  often  make  use  of  new  teehnologies.  As  noted  before,  when  the  systems 
eneounter  problems  at  sea,  fixing  them  beeome  eostly  beeause  of  the  large  expenses  that 
are  ineurred  when  SMEs  are  brought  from  land  to  ships  to  eorreet  the  problems.  The  idea 
of  E2E  testing  is  to  leverage  the  knowledge  and  existing  laboratories  on  the  eampus  (not 
aeross  the  eountry  as  in  the  DEP  testing  program)  and  within  a  eommand  (not  aeross 
different  eommands  as  in  the  DEP  testing  program)  to  ensure  that  C4I  teehnieal  solutions 
are  designed,  developed,  tested,  certified,  and  delivered  to  a  warship  before  its 
deployment  (Musson,  2008b).  In  addition,  not  only  does  the  E2E  testing  support  the 
eertification,  but  it  also  supports  the  developmental  testing  of  multiple  programs. 
Requested  by  PEO  C4I  to  provide  support  in  an  effort  to  eutting  sueh  expenses,  the  SSC- 
PAC  has  begun  to  build  an  E2E  lab  intended  to  replieate  and  test  shipboard  C4I  systems 
before  their  delivery  to  the  Eleet.  Building  the  E2E  lab  is  a  ehallenge  beeause  the 
sponsor  pays  only  a  fix  amount  of  funding.  The  E2E  testing  PM  must  therefore  analyze 
the  E2E  tasks  earefully,  identify  the  needs  the  E2E  lab  must  satisfy,  and  determine  the 
minimal  cost  to  build  the  E2E  lab. 

The  rest  of  the  chapter  is  organized  as  follows.  Section  B  discusses  the  E2E 
testing  eoneept;  Section  C  E2E  testing  program  eost;  Section  D  E2E  test  scheduling; 
Seetion  E  E2E  staffing,  and  Seetion  E  E2E  test  results.  Seetion  G  ends  with  a  summary 
of  the  ehapter. 
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B,  END-TO-END  TESTING  CONCEPT 

SSC-PAC  currently  manages  the  E2E  testing  program  and  the  building  of  the  E2E 
lab.  The  E2E  lab  is  to  house  as  mueh  C4I  hardware  as  possible  in  order  to  support  the 
E2E  testing.  Required  by  the  program  office  and  the  Eleet,  the  E2E  C4I  systems  shown 
in  Table  3  are  to  be  tested  in  the  E2E  testing  program  and  are  thus  to  be  housed  in  the 
E2E  lab  (Musson,  2008a)  . 

Eimited  in  funding,  the  E2E  testing  program  plans  to  employ  test  engineers  from 
the  PMWs  and  to  have  only  a  lab  manager,  a  lab  teehnieian,  network  engineers,  and 
systems  administrators  to  perform  day-to-day  lab  aetivities.  With  all  the  C4I  systems  in 
one  lab,  the  idle  time  observed  in  the  DEP  program  would  be  eliminated.  Envisioned  by 
the  PEO  C4I,  the  E2E  testing  eoneept  is  intended  (Musson,  2008b): 

•  To  ensure  the  eomplex  system-of-systems  capabilities  are  engineered,  tested 
and  eertified  to  work  together  in  a  eollaborative  environment  with 
transpareney  and  access  aeross  the  PEO  C4I  portfolio  of  solutions. 

•  To  support  the  development  of  serviee-oriented  arehitecture  enterprise,  in 
whieh  the  system  is  a  eollection  of  eomponents  and  services  developed  by 
multiple  programs. 

•  To  provide  aeeess  to  developers,  testers,  and  users  of  referenee 
implementations  of  systems/eomponents  and  to  make  sure  that  they  all 
interaet  together  without  having  to  proeure  them. 

•  To  provide  an  environment  for  eonfiguration  management,  base  lining 
interoperability,  functionality,  and  performance. 

•  To  have  the  E2E  lab  operated  by  an  E2E  system  engineering  team  and  to 
allow  events  to  be  staffed  by  PMs  and  stakeholders. 

•  To  have  the  E2E  lab  shared  by  multiple  PEO  C4I  sponsored  projeets. 

•  To  provide  mission  seenarios  and  test  strings  for  Unstruetured,  Structured, 
Pre-production,  and  Production  Environments  operating  at  multiple 
elassifications  (unclassified,  eonfidential,  and  seeret). 
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E2E  C4I  Systems 

Structured  Environment 

CFn 

MEDAL 

ADNS 

DMS 

SATCOM 

DCGS-N 

NITES 

ISNSCV) 

DMS  Proxy 

ADNS 

DMS 

Radiant  Mercury 

USWDSS 

GCCS-J 

ISNS(V) 

DMS  Proxy 

TBMCS 

AIS 

GCCS-J  w/B 

USWDSS 

GCCS-J 

NECC  (Small  Footprint) 

C2PC 

GCCS-M 

AIS 

GCCS-Jw/D 

Pre-Production 

CFn 

MEDAL 

C2PC 

GCCS-M 

SATCOM 

DCGS-N 

NITES 

Radiant  Mercury 

TBMCS 

NECC  (Small  Footprint) 

Table  3.  E2E  C4I  Systems  (Musson,  2008a) 

The  E2E  testing  PM,  interviewed  in  this  research,  believes  that  E2E  testing  will 
save  the  PEO  C4I  money.  The  reason  is  that  the  E2E  testing  will  reduce  the  probability 
of  having  at-sea  problems  with  the  deployed  systems  and  also  because,  if  the  problems 
arose  at-sea,  troubleshooting  might  not  be  as  costly  as  in  the  case  of  not  having  the  E2E 
testing  capability  since  the  problems  would  be  minor  ones.  The  E2E  lab  will  also  support 
the  E2E  testing  systems  engineering  concept  for  the  PEO  C4I  portfolio  enterprise 
engineering  such  as  development,  testing,  certification,  fielding,  and  sustainment.  The 
purposes  of  the  E2E  lab  are  thus  (Musson,  2008b): 

•  To  provide  program  transparency,  team/cross  team  collaboration,  and  to  speed 
up  delivery. 

•  To  reduce  integration  risk. 

•  To  provide  portfolio  block  and  build  recommendations  and  certifications. 

•  To  implement  a  distributed,  reconfigurable,  and  dynamic  lab  environment. 

•  To  allow  for  a  modular  design,  plug  and  play,  scheduled,  cost-effective 
method  of  testing. 

Additionally,  the  E2E  testing  program  is  envisioned  to  connect  its  C4I  systems 
(shown  in  Table  3)  at  SPAWAR  to  other  Navy  sites’  combat  systems,  such  as  NSWC 
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Dahlgren’s  Aegis  Weapon  Systems,  NWSC  DamNeek’s  ACDS  Bloek  0/1,  PAX  River’s 
Airborne  systems,  and  NSWC  Rhode  Island’s  Submarine  (Musson,  2008a). 


C,  E2E  TESTING  PROGRAM  COST 

A  number  of  programs  with  laboratory  spaees  need  to  move  to  different  loeations 
to  make  room  (spaee)  available  to  the  E2E  lab.  As  the  E2E  testing  program  has  to  pay 
for  the  moves,  the  total  eost  of  the  E2E  testing  program  inereases.  The  E2E  lab  was  to  be 
eompleted  in  2008  at  the  eost  of  EY08  $8.5M,  whieh  aeeounts  for  paying  the  labs  that 
need  to  move,  paying  eore  engineers,  buying  the  equipment,  and  building  the  E2E  lab 
(Williams,  2009).  More  money  is  now  needed  in  order  for  the  lab  to  be  eompletely  built. 
The  PEO  C4I  has  promised  to  provide  additional  funding  to  make  the  E2E  testing  of  the 
Battle  Group  effort  sueeessful,  but  the  amount  of  funding  has  not  yet  been  determined. 
The  eost  of  keeping  and  upgrading  the  E2E  lab  to  support  the  E2E  testing  initiative  ean 
potentially  be  high. 

The  E2E  testing  program  is  not  eurrently  in  the  Program  Objeetive  Memorandum 
(POM)  fund,  whieh  guarantees  funding  to  the  program.  Presently,  the  PEO  is  taxing  the 
PMWs  to  pay  for  the  E2E  testing  program  (Williams,  2009). 

D,  E2E  TESTING  SCHEDULING 

The  E2E  lab  should  be  available  for  all  on  a  first-eome,  first-serve  basis.  A  web- 
based  seheduling  system  has  been  ereated  to  allow  the  PM  or  Test  Direetor  to  sehedule 
his/her  tests.  Sinee  the  E2E  lab  is  still  under  eonstruetion,  the  availability  of  the  lab  has 
not  been  established  and,  therefore,  exeept  for  the  Eineoln  Battle  Group  test  re-seheduled 
in  late  PY09  by  the  request  of  PEO  C4I,  no  tests  have  been  seheduled  for  EY09 
(Williams,  2009). 

E,  E2E  TESTING  STAFFING 

A  lab  manager  will  manage  the  E2E  lab,  whieh  will  be  staffed  with  the  eore 
engineers  and  teehnieians  who  will  install,  maintain,  and  operate  the  lab  (Williams, 
2009). 
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F. 


E2E  LAB  TEST  RESULTS 


Since  the  E2E  lab  is  still  being  built,  exeept  for  the  eonneetivity  testing  between 
the  E2E  lab  and  the  other  labs  on  the  SSC-PAC  eampus,  no  full  tests  have  been 
eondueted.  The  eonneetivity  test  was  successful  (Aird,  2009). 

G.  SUMMARY 

In  summarize,  the  E2E  testing  eoneept  is  a  useful  eoneept  for  testing  C4I  systems 
at  the  Battle  Group  integration  level  before  delivering  them  to  the  Eleet  beeause  the  Navy 
potentially  saves  money  from  not  eondueting  testing  at  sea,  whieh  is  costly.  The  idea 
underlying  the  E2E  testing  eoneept  is  to  be  proactive  and  to  eorreet  potential  problems  in 
the  labs  in  order  to  reduee  issues  at  sea.  Aceordingly,  savings  will  be  made,  and  finaneial 
resourees  can  then  be  alloeated  to  support  other  efforts.  Being  able  to  integrate  all  the 
C4I  systems  in  the  lab  environment  would  be  the  first  aeeomplishment,  as  it  did  not  oeeur 
in  the  past.  The  future  will  tell  whether  this  eoneept  will  work  as  anticipated. 
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IV.  ANALYSIS 


A,  INTRODUCTION 

As  discussed  in  Chapters  II  and  III,  whereas  the  participant  sites  receive  funding 
on  a  per-test  basis  in  the  DEP  testing  program,  the  E2E  testing  program  does  not  pay 
engineers  from  the  PMWs  who  conduct  tests.  Also,  as  discussed  in  Chapter  III,  the  E2E 
lab  has  not  been  built  yet  because  management  did  not  have  a  complete  plan  to  establish 
the  lab.  This  chapter  discusses  the  results  of  a  qualitative  analysis  of  the  DEP  testing 
information  and  a  quantitative  analysis  of  the  DEP  testing  data  in  order  to  provide  data  to 
aid  the  DEP  testing  PM  in  making  decisions  on  the  paying  options.  It  also  demonstrates 
the  use  of  goal  programming  in  proving  data  to  support  the  creation  of  a  plan  for  building 
the  E2E  lab. 

The  rest  of  this  chapter  is  organized  as  follows.  Seetion  B  discusses  the  analysis 
of  the  DEP  testing  program  and  Seetion  C  the  E2E  testing  program.  Seetion  D 
summarizes  this  chapter. 

B,  DEP  TESTING  PROGRAM 

With  an  emphasis  in  the  area  of  ASW,  the  DEP  is  intended  to  support  only  testing 
of  Battle  Group  combat  systems.  Combat  systems  have  thus  far  been  the  subjects  of  DEP 
testing,  beeause  NAVSEA,  whieh  manages  not  only  the  DEP  program  but  also  the  Navy 
ships,  heavily  emphasizes  combat  systems  and,  in  particular,  safety  assoeiated  with  firing 
weapons.  As  the  combat  systems  are  tied  to  C4I  systems,  the  DEP  program  needs  to  add 
the  latter  systems  to  its  testing.  Strategically,  the  DEP  wants  to  expand  its  testing  to  C4I 
systems,  joint  sites  and  their  systems,  and  commercial  sites,  sueh  as  Northup  Grumman, 
Eockheed  Martin,  Boeing,  and  coalition  sites  and  their  systems.  However,  the  current 
DEP  budget  might  not  be  able  to  support  this  expansion  (Coyne,  2006).  Also,  the 
expansion  to  the  eommercial  sites  has  not  materialized  because  the  DEP  program  fails  to 
justify  the  value  added  by  the  commercial  sites.  No  new  sites  have  been  added.  Even  if 
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the  new  sites  were  needed,  funding  from  the  sponsors  (NAVSEA  and  OPNAV)  would 
not  be  available  to  support  the  additions.  The  eore  sites  discussed  in  Chapter  II  have  thus 
remained  in  use. 

As  pointed  out  in  Chapter  II,  the  DEP  participant  sites  are  paid  on  a  per-test  basis 
(Mann,  2004).  This  pay-per-test  concept  has  led  to  a  major  problem  for  the  DEP 
program,  which  is  the  time  loss  at  the  beginning  of  a  test.  This  time  loss,  meaning  the 
time  not  used  for  carrying  out  the  test,  is  caused  by  the  high  personnel  turnover  rate, 
which  in  turn  is  a  consequence  of  the  pay-per-test  concept.  A  site  that  is  to  carry  out  a 
test  will  hire  part-time  engineers  to  do  the  testing.  Upon  completing  the  test,  these 
engineers,  who  are  no  longer  funded,  move  on  to  other  opportunities  or  to  take  full-time 
positions  elsewhere.  The  same  phenomenon  occurs  test  after  test,  resulting  in  a  high 
personnel  turnover  rate.  When  either  a  test  need  be  repeated  or  a  new  test  need  be 
performed,  newly  hired  engineers,  who  have  little  or  no  knowledge  of  the  program,  spend 
time  to  learn  and  to  set  up  the  test,  resulting  in  an  ineffective  use  of  the  allocated  testing 
time.  Eurthermore,  the  time  loss  is  also  caused  by  a  few  sites  having  problems,  as  the  rest 
of  the  sites  have  to  wait  until  the  troubled  sites  are  fixed  before  all  the  sites  can  carry  out 
the  testing. 

An  analysis  of  the  data  captured  from  the  reviewing  of  the  DEP  test  reports 
supports  the  findings  described  immediately  above.  The  data  refer  to  the  uptimes  and 
downtimes  of  the  sites,  collected  between  PY02  to  PY06  (OTHT,  2008),  shown  in 
Appendix  A.  The  analysis  involves  the  calculation  and  an  examination  of  the  mean  times 
between  failures  (MTBE)  of  the  sites  during  this  period.  The  MTBE,  which  is  defined  to 
be  the  total  of  downtime  minus  the  total  of  uptime  divided  by  the  number  of  failures 
during  the  indicated  period,  is  calculated  according  to  (Wikipedia). 


MTBF  =  ^ 


Downtime  -  Uptime 
#of  Failures 


Based  on  the  data  in  Appendix  A,  the  calculated  MTBE  equals  to  3  hours  and  6 
minutes,  which  is  roughly  38%  of  an  eight-hour  test  day.  The  DEP  program  needs  to 
reduce  or  eliminate  this  large  lost  time.  The  outcomes  of  the  interview  with  the  lead 
(Tran,  2009)  indicate  that  the  high  turnover  rate  of  the  testers  at  the  sites,  coupled  with 
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new  personnel  that  are  not  familiar  with  the  DEP  testing  environment,  contribute  to  the 
substantial  amount  of  time  lost  per  test  day.  The  reasons  for  this  high  personnel  turnover 
are; 

•  Engineers  are  not  fully  funded  and  cannot  find  a  part-time  project  to  make  up 
for  the  short  fall. 

•  It  is  difficult  to  split  the  task  between  two  or  more  efforts. 

•  Testers  unfamiliar  with  the  test  cannot  get  help  since  the  test  is  conducted 
after  hours  when  the  experts  have  already  left  for  the  day. 

•  Eormal  Risk  Mitigation  Test  (formally  called  a  dry  run)  has  been  scaled  down 
from  one  week  to  one  day,  and  then  to  none,  in  order  to  save  money. 

Eigure  2,  which  displays  the  delays  for  the  tests  repeated  during  the  EY02  to 
EY06  periods,  indicates  that  the  first  day  of  each  test  is  the  most  troublesome  day  of  the 
entire  test  period,  as  it  takes  more  than  two  hours  of  every  8-hour  test  day  to  get  the 
systems  ready,  which  would  be  unacceptable  for  any  test.  As  a  mandatory  requirement 
for  the  sites,  the  amount  of  set-up  time  must  be  reduced  to  less  than  an  hour  per  test  day 
to  complete  the  necessary  tests. 

The  DEP  program  has  been  operating  since  1999,  yet  the  PM  continues  to  have 
difficulties  in  determining  whether  the  program  should  pay  the  participant  sites  full-time 
funding  or  per-test  amounts.  The  quantitative  analysis,  addressed  in  Chapter  V,  is  used  to 
aid  the  DEP  PM  in  determining  an  effective  paying  option. 

Eigure  2  also  shows  that  in  some  cases  the  test  day  is  completely  lost,  as  reflected 
by  the  peaks  of  the  corresponding  curves  (e.g.,  EY02,  Test  1,  and  day  8).  With  the  lost 
time  occurring  almost  daily,  the  DEP  PM  should  conduct  a  risk  mitigation  effort  to  figure 
out  what  the  sites  can  do  to  reduce  the  lost  time.  The  management  should  be  held 
accountable  for  not  conducting  such  a  risk  mitigation  effort.  If  one  site  were  unable  to 
support  testing,  then  DEP  would  be  able  to  utilize  the  remaining  sites. 
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C.  END-TO-END  TESTING  PROGRAM 

Again,  contrary  to  the  DEP  program,  the  E2E  testing  program  is  concentrated 
mainly  on  C4I  systems.  The  E2E  testing  program  emphasizes  the  area  of  serviees,  such 
as  Domain  Name  Server  (DNS)/EmaiEWeb;  communieation  systems,  sueh  as 
Consolidated  Afloat  Networks  and  Enterprise  Serviees  (CANES)/Advanced  Digital 
Network  System  (ADNS)/Teleport/Network  Operation  Center  (NOC);  Information 
Assurance  (lA),  sueh  as  Gold  DislCVirtual  Private  Network  (VPN)/Cryptos;  and  network, 
such  as  Satellite  Communications  (SATCOMs)  &  Line  of  Sight  (LOS)  &  Pierside.  In  the 
future,  E2E  testing  may  be  expanded  to  include  systems  outside  of  the  C4I  arena  (E2E, 
2008). 
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As  mentioned  in  Chapter  III,  the  E2E  testing  has  planned  to  use  testers  from 
PMWs.  The  deeision  to  leverage  the  PMW  testers  is  flawed  beeause  (Aird,  2009): 

•  The  PMW  testers  would  leave  after  eompletion  of  a  test  and  would  also  take 
with  them  their  knowledge,  whieh  might  not  be  available  when  needed  for 
additional  testing.  If  problems  were  diseovered  from  a  previous  test,  the 
PMW  testers  would  not  be  around  to  help  with  the  reereation  of  the  problems. 
Even  if  they  eould,  the  PMW  testers  would  not  be  able  to  help  immediately 
sinee  they  might  have  already  been  working  on  different  tasks. 

•  In  regards  to  the  expertise  of  the  testers,  if  the  testers  were  not  seasoned 
testers,  then  the  E2E  testing  of  C4I  systems  probably  would  eneounter  the 
time  loss  problems  faced  by  the  DEP  testing  program. 

•  Eeveraging  testers  from  different  programs  would  bring  a  lack  of 
accountability  as  each  program  would  claim  effectiveness  when,  in  actuality, 
history  has  shown  otherwise.  Also,  in  the  case  of  failure,  a  program  would 
blame  another  program.  Eurthermore,  working  across  many  programs 
requires  timely  coordination  and  flexibility  of  schedule  in  order  to  support  the 
E2E  testing  program.  In  addition,  gathering  all  testers  from  different 
programs  at  one  time  would  not  be  an  easy  task  and  an  even  harder  task  when 
the  schedules  slip  to  the  right  (Aird,  2009). 

A  funding  issue  arises.  The  PEO  C4I  forces  the  PMWs  to  support  the  E2E  testing 
program  and  taxes  their  programs  to  pay  the  E2E  testing.  The  PMWs,  however,  argue 
that  their  testing  capability  would  be  sufficient  (Aird,  2009)  to  support  the  Elects  and, 
being  short  on  personnel  to  support  additional  tasks,  want  their  programs  to  be  left  alone. 
The  PMWs  also  believe  that  the  E2E  testing  program  should  be  POM  funded,  rather  than 
being  paid  by  taxing  the  PMWs. 

Eurthermore,  future  funding  secured  from  taxing  the  PMWs  could  not  always  be 
certain.  The  Assistant  E2E  testing  PM  is  not  sure  of  how  much  funding  or  if  any  funding 
will  be  available  for  EYIO  (Williams,  2009).  With  the  funding  uncertainty,  the  E2E 
testing  program  might  not  be  sustainable  in  the  future.  Eor  example,  the  Horizontal 
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Integration  program  from  SPAWAR,  which  had  been  funded  from  taxing  PMWs,  was 
terminated  upon  Adm.  Gauss’  retirement  (Aird,  2009). 

The  results  of  the  interview  with  the  E2E  testing  PM  (Williams,  2009)  reflect  the 
uncertainties  the  PM  has  experienced  in  regards  to  building  the  E2E  lab  with  the 
available  funding.  The  first  E2E  testing  of  the  Eincoln  Battle  Group,  which  was  to  occur 
in  December  2008  (E2E,  2008),  did  not  take  place,  because  the  E2E  lab  was  not  ready. 
In  fact,  the  E2E  lab  is  currently  not  ready  to  support  any  testing,  and,  because  of  funding 
uncertainty,  the  PM  does  not  know  when  it  will  be  up  and  running.  In  addition,  the  PM 
does  not  have  a  complete  plan  to  establish  the  E2E  lab,  which  has  contributed  to  the 
delay  in  getting  the  E2E  lab  ready.  The  goal  programming  approach,  addressed  in 
Chapter  V,  is  used  to  aid  the  E2E  testing  PM  in  determining  an  effective  implementation 
plan  to  build  the  lab. 

D.  SUMMARY 

Prom  the  review  of  the  existing  literature  on  the  DEP  and  E2E  testing  programs 
and  from  the  interviews  with  the  respective  PMs  and  leads,  (1)  the  DEP  pay-per-test 
concept  has  led  to  a  major  problem  of  testing  inefficiency,  namely,  the  loss  of  test  time  to 
setting  up  the  labs  by  inexperienced  part-time  engineers,  and  (2)  the  E2E  testing  program 
does  not  have  a  complete  plan  to  build  the  E2E  lab. 
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V.  COMPARATIVE  ANALYSIS  AND  GOAL  PROGRAMMING 


A,  INTRODUCTION 

As  noted  in  Chapter  II,  the  DEP  program  has  been  operating  since  1999,  yet  the 
PM  continues  to  have  difficulties  in  determining  whether  the  program  should  pay  the 
participant  sites  full-time  funding  or  per-test  amounts.  Rather  than  paying  for  full-time 
testers,  the  DEP  PM  has  chosen  the  pay-per-test  method.  The  comparative  analysis 
discussed  in  this  chapter  aims  at  providing  data  to  aid  the  DEP  PM  in  selecting  the  best 
paying  option  for  the  DEP  testing  program. 

As  observed  in  Chapter  III,  the  E2E  lab  has  not  been  built.  One  of  the  causes  of 
this  debacle  is  the  E2E  testing  PM  has  failed  to  determine  the  funding  needed  to  build  the 
E2E  lab  and  to  get  it  ready  to  support  testing.  The  goals  for  E2E  testing  have  not  been 
established  and  spelled  out  clearly  in  the  plan  to  build  the  E2E  lab.  To  remedy  this 
undesirable  situation,  this  thesis  uses  goal  programming  (GP)  (Ragsdale,  2007)  to 
establish  the  goals  of  the  E2E  lab  in  the  presence  of  funding  constraints. 

The  rest  of  this  chapter  is  organized  as  follows.  Section  B  presents  the  results  of 
the  comparative  analysis  of  the  two  paying  approach:  pay-per-test  and  pay-full-time. 
Section  C  discusses  and  demonstrates  the  goal  programming  approach  to  aiding  in  the 
creation  of  a  plant  for  building  the  E2E  lab.  Section  D  provides  a  brief  summary  of  the 
chapter. 

B,  COMPARATIVE  ANALYSIS 

Table  4  shows  seven  distributed  test  sites  involved  in  interoperability  certification 
testing:  Dahlgren,  Dam  Neck,  NSWC  San  Diego,  NAVAIR,  SSC-PAC  TADIE,  SSC- 
PAC  GCCS-M,  and  Wallops  Island.  As  discussed  in  Chapter  II,  NAVSEA  has  two 
paying  options  for  the  DEP  testing:  paying  the  sites  a  fixed  amount  of  money  to  retain 
full-time  testers  or  paying  the  sites  on  a  per-test  basis.  The  first  option  is  not  necessarily 
cost-effective,  as  the  sites  will  constantly  deplete  resources,  specifically  funding,  whether 
they  are  doing  testing,  or  staying  idle.  The  latter  paying  option  might  save  NAVSEA 


29 


money,  but  as  discussed  in  Chapter  IV,  it  results  in  38%  of  the  testing  lost  to  the  initial 
test  set-up.  Constrained  by  the  available  budget,  the  DEP  program  must  decide  between 
the  two  paying  options.  To  aid  the  DEP  program  in  making  an  informed  decision,  a 
comparative  analysis  is  performed  this  research,  in  which  the  DEP  data  are  collected  and 
analyzed,  to  answer  the  question  as  to  which  pay  method — pay-per-test  or  pay-full- 
time — the  DEP  program  should  use. 

The  comparison  is  made  for  all  the  years  of  DEP  testing  from  EYOl  to  PY09. 
Eigure  3  shows  the  number  of  tests  carried  out  in  those  years,  which  ranges  from  three  to 
eight  per  year.  It  is  noted  that  less  than  six  tests  were  carried  out  in  EYOl  and  EY05,  and 
at  least  six  tests  were  carried  in  the  remaining  years.  Eor  the  purpose  of  illustration, 
EY09,  during  which  eight  tests  are  conducted,  is  considered  for  comparing  the  costs 
resulting  from  using  the  two  different  methods  of  payment.  The  EY09  data  shown  in 
Table  4  are  input  to  the  comparative  analysis.  The  data  (BEIMS,  2009)  are: 

•  The  maximum  number  of  engineers  at  a  test  site,  which  is  two,  to  support  a 
test  from  pre-test  through  the  post-test,  and  their  hourly  salary. 

•  The  total  test  time  of  two  weeks,  implying  160  hours  for  two  engineers. 

•  The  time  allocated  for  attending  test  planning  working  group  meetings 
(TPWG),  working  on  test  plan/procedures,  and  reviewing  test 
plan/procedures,  which  totals  two  and  half  weeks,  implying  200  hours  for 
two  engineers. 

•  The  time  for  pre-test  checkout  prior  to  the  actual  test,  which  is  32  hours  for 
two  engineers. 

•  The  time  allocated  for  post- test  draft/final  test  report  and  trouble  reports  input 
to  the  database,  which  is  two  weeks,  implying  160  hours  for  two  engineers. 

•  The  total  lost  time  per  test  incurred  by  two  engineers,  which  varies  from  test 
to  test. 

•  The  total  man-hours  per  year,  which  is  1750. 
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mVSEADEPPROGRANI 


FY09  Data 

NSVCDD 

NSVC  DN 

ICSTD  SD 

NAVAIR 

TADILSD 

GCCS-MSD 

NSVC  VI 

Total  Hourlj 
Cost  lot  All 

Sites  (S) 

Enqineei  1  (Cost  Pei  Hour) 

SI20 

S11S 

S10S 

?S0 

Enqineei  2  (Cost  Pei  Hour) 

S12S 

S10S 

S11S 

S10S 

StIS 

sits 

SIOS 

?ss 

Hours 

Enpineei  1 
Cost  ISl 

Enpineei  2 
Cost  ISl 

Total  Cost 

Test  Time  (2  VeeksKHouis) 

SO 

TPVG  (2  »2  Weeks)  (Homs) 

ISO 

Pie-Test  Checkout  (2  Dajs) 
(Hours) 

IS 

Post  Test  Data  InputiReport  (2 
Veeks)(Houis) 

SO 

Total  Hours  Pei-Test 

2?S 

Houili  Cost  (i) 

?S0 

?SS 

Total  Cost  Per-TesI  (without 
Lost  Time)  ($) 

21S2S0 

2tSSS0 

S4SI.940 

Maiimum  Lost  Time  Pei-Test 

Test! 

24 

1S?20 

tSS40 

SST.SSO 

Test  2 

SI 

24IS0 

24SSS 

S4S.S1S 

Tests 

29 

22S20 

22?SS 

S4S.SSS 

Test  4 

S? 

2SSS0 

2904S 

SS?.90S 

Tests 

2S 

I9S00 

19S25 

IS9.I2S 

Tests 

2? 

210S0 

2I19S 

S42.2SS 

Test? 

14 

10920 

10990 

S21.910 

Tests 

IS 

124S0 

12SS0 

S2S.040 

Total  Cost  Pei-Test  (Plus  Lost 
Time) 

Total  Paj  Full- 
Time  Option 

Testi 

S4S9.S00 

S4S9.S00 

S2.?SS.?S0 

Test  2 

S4S0.4SS 

S949.9SS 

S2.?SS.?S0 

Tests 

S4??.S2S 

SI.42?.2S0 

S2.?SS.?S0 

Test  4 

S4S9.S4S 

S1.91?.12S 

S2.?SS.?S0 

Tests 

S4TI.0SS 

S2.SSS.I90 

S2.?SS.?S0 

Tests 

S4T4.19S 

S2.SS2.SSS 

S2.?SS.?S0 

Test? 

S4SS.SS0 

SS.SIS.2SS 

S2.?SS.?S0 

Tests 

S4SS.9S0 

SS.??S.21S 

S2.?SS.?S0 

S  Tests  Total  (FYOS  Total) 

SS.??9.21S 

Man  Year  Hours  (1?50) 

l?S0 

ISSSOOO 

1S?S?S0 

S2.?SS.?S0 

Table  4.  NAVSEA  DEP  Program  PY09  Data 
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#  of  Tests  Conducted  From  FY01-FY09 


■  #  of  Tests 


2001  2002  2003  2004  2005  2006  2007  2008  2009 


Fiscal  Year 

Figure  3.  Number  of  Tests  Conducted  From  FY01-FY09 

A  calculation  using  the  data  in  Table  4  shows  that,  for  conducting  eight  tests 
during  FY09  with  two  engineers,  using  the  pay-per-test  method  costs  approximately 
$3.8M  whereas  using  the  full-time  payment  method  costs  about  $2.7M 

The  same  calculation  is  carried  out  to  obtain  the  total  costs  from  FYOl  to  FY09 
(BFIMS,  2009),  using  all  but  the  hourly  costs  in  Table  5  and  accounting  for  the  different 
number  of  tests  from  year  to  year.  Figure  4,  displaying  the  costs  as  a  function  of  the 
number  of  tests  incurred  from  using  the  two  paying  methods,  indicates  that  if  the  number 
of  tests  per  year  is  not  greater  than  six,  the  pay-per-test  method  is  cheaper  than  the  pay- 
full-time  option.  If  more  than  six  tests  per  year  are  carried  out,  the  full-time  is  cheaper 
than  the  pay-per-test  option.  Note  that  the  comparison  is  made  with  respect  to  cost  only. 
Factors  other  than  cost  that  may  play  in  the  decision  making  process  are  not  considered  in 
this  thesis. 

Furthermore,  as  six  or  less  tests  are  conducted  for  only  two  out  of  the  nine  years, 
the  pay- full-time  option  would  have  collectively  saved  money  during  FY01-FY09.  In 
this  case,  the  core  experienced  engineers  would  have  been  likely  retained  and  the  lost 
time  would  have  been  reduced,  if  not  avoided.  Based  on  Figure  4,  the  DEP  program 
would  have  saved  roughly  $1.2  M  over  the  FY01-FY09  if  the  pay-full-time  option  had 
been  instituted.  Thus,  in  the  long  run,  if  more  than  six  tests  are  conducted,  savings  will 
be  made  with  the  pay-full-time  option. 
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’ay-Per-Test  and  Pay-Full-Time  i 

!]omparison 

FISCAL 

YEAR 

2001 

2002 

2005 

2006 

2007 

2000 

2009 

PAY-PER-TEST  OPTION 

tOfTtst 
Coidicttd 
P<i  Y»i 

4 

6 

7 

6 

3 

8 

7 

6 

8 

PER-TEST 
FOR  ALL 

SITES 

Havrly  Cvl  Far 
CMtld) 

SIT 

545 
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635 

663 

704 

741 
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Haarl^  Carl  Far 
EaitZCt) 

S21 
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708 
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785 

fatal  Haar/  far 

276 
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276 
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276 

276 

fatal  Haar#  far 
Eai  tZ  (Haar/) 
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276 
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276 

276 

276 

fatal  Vart  far 

Eaatl  (Vitkaal 
Lartfi.*)  ($) 

142.821 

150.338 

158,251 
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184.576 

134,230 

204.516 

215,280 

fatal  Cart  Far 
Caat2 
(tfitkaat  Lart 

143,737 

151,302 

176,471 

185,753 

135,536 

205,827 

216,660 

fatal  Cart  far  2 
Ea^iaaarr 
(tfitkaat  Lart 

286.SSS 

301.640 

317.516 

334,227 

351,818 

370,335 

383,826 

410.343 

431,340 

fatal  Has  Lart 
fiB*  Haarr  far 
Eack  Eaaiaaarr 

35.3 

31.0 

13.3 

24.5 

23.7 

15.6 

13.6 

22.0 

25.4 

fatal  Lart  Has 
f  Cart  far 

18,241 

16,886 

11,058 

14,787 

15,036 

10,443 

13,777 

16,302 

13,733 

fatal  Has  Lart 

f  IB*  Cart  far 

18,358 

16,334 

11,123 

14,882 

15,132 

10,516 

13.866 

16,407 

13,313 

fatal  Has  Lart 
f  IB*  Cart  far  2 
(.,!»•»(!) 

36,538 

33,880 

22,187 

23,663 

30,168 

20,365 

27,643 

32,703 

33,712 

fatal  Cart  far  2 
Ea^iaaarr  (tfitk 

L»tfiB*)(t) 

323.156 

335,520 

333.702 

363.836 

381,386 

331,300 

417.463 

443,052 

471.652 

PER  YEAR 

Toul  Cost 
Pti  Ymi  for  2 
Etqitttis  ft) 

t1.232.625 
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t2.377.316 

t2.183.3T4 

t1.145.35T 

t3.130.400 

t2.322.281 

t2.658.303 

t3.773.215 

PAY-FULL-TIME  OPTION 

NaBk*r  af 
Haarr  Par  T*ar 
»*r  Eaaia**r 

1,750 

1,750 

1,750 

1,750 

1,750 

1,750 

1,750 

1,750 

1,750 

Eaaia**ri1 
Hairir  Cart  (t) 

517 

545 

573 

604 

635 

663 

704 

741 

780 

Eaaiaaar  #2 

Haarly  Cart  (t) 

521 

548 

577 

607 

633 

673 

708 

746 

785 

fatal  Cart  Far 

EMil(l) 

304,750 

353,230 

1,003,400 

1,056,211 

1,111,801 

1,170,317 

1,231,313 

1,236,750 

1,365,000 

fatal  Cart  Far 
Eaat2(|) 

311,750 

353.341 

1,003.832 

1.062,382 

1,118,328 

1,177.813 

1.233.803 

1.305,063 

1,373,750 

ToUl  Cost  for 
2  Ekgikttrs 
Ptr  Year  (I) 

t2,013.233 

t2.113.133 

t2.230,723 

t2.348.136 

t2.471.T22 

t2,601.813 

t2.T38,T50 

Table  5.  FY01-FY09  Pay-per-Test  vs.  Pay-Full-Time 
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Figure  4.  FY01-FY09  Pay-per-Test  vs.  Pay-Full-Time 

C.  E2E  TESTING  PROGRAM 

The  E2E  lab,  whieh  was  to  conduct  testing  of  a  battle  group  by  November  2008, 
is  still  in  the  planning  phase.  The  E2E  testing  program  is  still  determining  the  funding 
needed  to  build  the  E2E  lab  and  to  get  it  ready  to  support  testing.  The  funding  issue  is 
elaborated  in  Chapter  III.  The  level  of  funding  depends  on  the  needs  (or  goals)  of  the 
E2E  lab,  which,  as  part  of  the  E2E  lab  building  plan,  must  be  determined  and  supported 
by  a  rigorous  analysis. 

As  goals,  the  lab  needs  to  have  desk/chairs,  racks,  conference  rooms,  and  space. 
Specifically,  for  the  purpose  of  illustration,  the  E2E  lab  building  scenario  considered  in 
this  thesis  requires  approximately  40  desks  and  chairs,  20  computer  racks,  and  2 
conference  rooms.  The  desks/chairs,  racks,  conference  rooms,  and  the  lab  require  100, 
100,  1,050,  and  8,000  ft^oi  space,  respectively.  Eurthermore,  the  desks/chairs,  racks,  and 
conference  rooms  each  cost  $5,000,  $7,000,  and  $156,000,  respectively.  The  budget 
available  for  the  E2E  lab  is  $650,000. 

Regarding  meeting  these  goals,  underachieving  the  first  three  goals  related  to  the 
number  of  desks/chairs,  racks,  conference  rooms  would  be  undesirable,  but 

overachieving  these  goals  would  be  acceptable.  Also,  underachieving  or  overachieving 
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the  goal  of  adding  8,000  ft^  would  be  undesirable.  Finally,  only  spending  more  than 
$650,000  would  be  undesirable.  (The  objeetive  funetion  defined  below  refleets  this 
eonsideration.) 

The  question  to  answer  is:  Can  the  E2E  testing  program  deliver  the  E2E  lab  that 
will  meet  these  goals  for  sueh  a  budget?  The  approaeh  to  answering  this  question  is  to 
formulate  the  E2E  testing  optimal  planning  problem  as  a  goal  programming  problem  and 
to  solve  the  resulting  goal  programming  problem  using  Exeel  Solver. 

The  E2E  testing  goal  programming  problem  is  now  formulated.  Eet  stand  for 
the  number  of  desks  and  ehairs,  the  number  of  eomputer  raeks,  and  and  the 
number  of  conferenee  rooms.  Eet  vj ,  v^,  V3 ,  V4 ,  and  vj  be,  respeetively,  the  target 
values  of  the  number  of  desks  and  chairs,  the  number  of  computer  racks,  the  number  of 
conference  rooms,  the  total  square  footage,  and  the  total  cost,  which  are  40,  20,  2, 
8,000  ft^ ,  and  $650,000,  respectively.  These  values  are  not  derived  from  a  rigorous 
analysis,  and  they  might  not  be  possibly  met.  To  account  for  this  unfounded 
restrictiveness,  let  the  so-called  deviational  variables  dTaxid  ,  ;  =  l,---,5,  represent  the 

amounts  by  which  the  goals  can  deviate  from  their  respective  target  values.  Specifically, 
represents  the  amount  by  which  the  goal’s  target  value  is  underachieved,  and 

represents  the  amount  by  which  the  goal’s  target  value  is  overachieved. 

The  constraints  are  thus: 

+fl',-2T"2  +«,3T"3  -d^  =  Vi 

X.  e  Z  and  X.  >  0 
dl,dt>Q 

in  which,  for  the  E2E  testing  problem  at  hand,  the  coefficients  =  «,2  =  «,3  =  1 , 
«,4=«,5=0,  for  ;  =  l,2,and3,  and  «4i  =  «42  =  100/?^ ,  0:43  =  1,050^^^ ,  and 

=  $5,000,0:52  =  $7,000,  and  «53  =  $156,000.  Z  denotes  the  set  of  integers; 
X.  G  Z  andX.  >  0  together  mean  that  X.  are  nonnegative  integers. 

The  objective  function  is  defined  as 
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z  = 


in  which  the  weight  eoeffieients,  w  ,i  =  l,  -,5,  represent  numerie  eonstants  that  ean 

be  assigned  values  to  weight  the  various  deviational  variables.  The  objeetive  funetion  is 
thus  a  weighted  pereentage  deviation.  A  variable  that  represents  a  highly  undesirable 
deviation  from  a  partieular  goal  is  assigned  a  relatively  large  weight.  A  variable  that 
represents  a  neutral  or  desirable  deviation  from  a  partieular  goal  is  assigned  a  weight  of  0 
or  lower  than  0.  Analysis  of  the  goal  programming  begins  with 
=  wj  =  W3  =  W4  =  W4  =  =  1  and  all  other  weights  being  0. 

The  E2E  testing  goal  programming  problem  is  thus;  Minimize  z  subjeet  to  the 
eonstraints  above.  The  solution  to  this  E2E  testing  goal  programming  problem  is 
obtained,  using  Exeel  Solver  (Ragsdale,  2007),  and  is  eaptured  in  Table  6. 

As  indieated  by  Table  6,  the  planned  eost  (budget),  whieh  is  $650,000,  exeeeds 
the  optimal  eost,  whieh  is  $647,000,  by  $3,000.  Whereas  money  savings  are  realized,  the 
targeted  number  of  desks  and  ehairs  is  short  by  one.  As  aforementioned,  this  negligible 
shortage  is  aeeeptable. 


E2E  Goal  Programming 

Problem  Data 

Desk  &  Chair 

Rack 

Conference  Room 

Square  Footage 

100 

100 

1050 

Cost 

5000 

7000 

156000 

Goal  Contraints 

Desk  &  Chair 

Rack 

Conference  Room 

Sq.  Ft. 

Cost 

Actual  Amount 

39 

20 

2 

8000 

$647,000 

Under  + 

1 

0 

0 

0 

$3,000 

Over- 

0 

0 

0 

0 

0 

Goal  = 

40 

20 

2 

8000 

$650,000 

Target  Value 

40 

20 

2 

8000 

$650,000 

Table  6.  E2E  Testing  Goal  Programming  Solver  Results 
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The  E2E  testing  PM  ean  thus  use  this  formulated  problem  with  different  budgets 
and  numbers  of  items,  sueh  as  desk/ehairs,  raeks,  eonferenee  rooms,  and  spaee,  ete.,  to 
eome  up  with  the  best  available  minimum  eost  to  support  the  E2E  testing  effort. 
Additionally,  the  E2E  testing  PM  ean  use  this  method  to  justify  the  funding  for  the 
eurrent  year  and  future  funding. 

D,  SUMMARY 

In  summary,  the  eomparative  analysis  indieates  that  paying  the  DEP  sites  full¬ 
time  will  save  the  DEP  program  if  it  eonduets  six  or  more  tests  per  year.  If  the  number  of 
tests  is  less  than  six  per  year,  then  DEP  program  has  the  flexibility  in  seleeting  either  of 
the  paying  options.  An  additional  advantage  of  paying  the  DEP  sites  on  a  full-time  basis 
is  that  the  DEP  sites  will  likely  be  able  to  retain  experieneed  testers  for  a  long  period  of 
time  and  henee  to  be  able  to  keep  the  eontinuity  and  eohesiveness  of  the  team. 

Goal  programming  ean  be  used  effeetively  to  aid  in  establishing  a  plan  for 
building  an  E2E  lab  for  the  Navy  E2E  testing.  It  provides  a  rigorous  approaeh  to 
determining  the  goals  of  the  E2E  lab  in  a  timely  fashion  (henee,  saving  money)  that  ean 
meet  budgetary  eonstraints.  It  ean  also  be  used  in  support  of  planning  for  programs  other 
than  testing  as  well  as  for  many  multi-goal  problems  eneountered  in  systems  engineering. 
Results  obtained  with  this  goal  programming  approaeh  are  used  in  deeision  making,  in 
assessing  the  feasibility  of  aehieving  the  goals,  and  also  in  knowing  how  mueh  funding  is 
required  to  meet  the  goals. 
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VI.  CONCLUSION 


Dealing  with  test  and  evaluation  of  combat  and  C4I  systems  in  the  U.S.  Navy,  this 
thesis  focuses  specifically  on  two  separate  Navy  testing  programs  managed  by  two 
different  commands;  the  DEP  testing  program,  run  by  NAVSEA,  and  the  E2E  testing 
program,  currently  being  formed  by  SPAWAR. 

During  deployments  and  Battle  Group  exercises.  Battle  Group  systems 
encountered  interoperability  problems  such  as  erroneous  dual/multiple  track  designations, 
misidentification/track  identity  conflicts,  report  responsibility  conflicts,  friendly  tracks 
displayed  as  unknown/pending,  tracks  dropped  without  operator  action,  different  track 
identities  of  different  ships,  etc.  When  these  interoperability  problems  occurred  at  sea, 
fixing  them  was  costly  because  of  the  high  cost  incurred  in  bringing  SMEs  from  land  to 
ships  to  correct  the  problems.  In  an  effort  to  prevent  these  costly  problems,  the  DEP 
testing  program  was  formed  in  June  1998  to  support  the  evaluation  of  the  interoperability 
of  Battle  Group  systems  and  also  to  aid  in  design  decision  making  early  in  the  systems 
acquisition  process.  The  DEP  consists  of  shore-based  combat  system  sites,  which  are 
currently  paid  on  a  per- test  basis,  rather  than  on  a  retaining  full-time  basis. 

The  purpose  of  the  E2E  testing  is  to  evaluate  integrated  capabilities  of  shipboard 
C4I  systems  for  interoperability.  The  shipboard  C4I  systems  need  be  tested  before  their 
delivery  to  the  warships.  Not  only  does  E2E  testing  support  the  certification,  but  it  also 
supports  the  developmental  testing  of  multiple  programs.  In  the  future,  the  E2E  testing 
may  be  expanded  to  include  systems  outside  of  the  C4I  arena.  In  the  E2E  testing 
concept,  formulated  in  2007,  a  single  E2E  systems  engineering  laboratory,  which  is  yet 
to  be  built,  replicates  and  tests  these  C4I  systems.  Eimited  in  funding,  the  E2E  testing 
program  plans  to  employ  test  engineers  from  the  PMW  organizations.  A  challenge  faced 
by  the  E2E  testing  program  is  building  and  getting  the  E2E  lab  ready  for  testing.  Two 
factors  contributing  to  this  challenge  are  uncertain  availability  of  funding  for  building  the 
E2E  lab  and  the  lack  of  a  comprehensive  plan  to  establish  the  E2E  lab. 

With  a  focus  on  the  issue  of  cost-effective  implementation  of  the  DEP  and  E2E 
testing  programs,  the  research  performed  in  this  thesis  involves  (1)  conducting  a  review 
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of  the  past  and  current  test  results  of  the  distributed,  E2E  programs  and  their  archived 
documents,  related  distributed  and  E2E  testing  studies,  and  other  pertinent  related 
material  such  as  test  report,  test  plans,  and  test  procedures;  (2)  developing  interview 
questionnaires  directed  at  managers  of  the  DEP  and  E2E  testing  programs,  (3)  conducting 
interviews  with  the  DEP  and  E2E  testing  PMs,  test  directors,  and  functional  leads  and 
process  literature  and  interview  results;  and  (4)  performing  qualitative  and  quantitative 
analysis  and  goal  programming  to  aid  in  determining  the  optimal  costs  of  carrying  out 
these  testing  methods.  The  findings  from  this  research  follow. 

The  DEP  program  currently  pays  the  DEP  participant  sites  on  a  per-test  basis.  As 
a  result,  the  allocated  testing  time  is  not  fully  used  for  testing;  roughly  38%  of  the 
allocated  testing  time  is  lost  at  the  beginning  of  a  test.  Another  paying  option  for  the 
DEP  program  to  consider  is  paying  the  DEP  participant  sites  a  fixed  amount  of  money  to 
retain  full-time  engineers.  The  testing  cost  data  from  EYOl  to  PY09  indicate  that 
conducting  eight  tests  during  PY09  with  two  engineers,  using  the  pay-per-test  method 
costs  approximately  $3.8M,  whereas  using  the  full-time  payment  method  costs  about 
$2.7M,  and  that  if  the  number  of  tests  per  year  is  not  greater  than  six,  the  pay-per-test 
method  is  cheaper  than  the  pay-full-time  option.  As  six  or  less  tests  were  conducted  for 
only  two  out  of  the  nine  years  (EYOl  and  EY05),  the  pay- full-time  option  would  have 
collectively  saved  money  during  EY01-EY09.  Thus,  in  the  long  run,  if  more  than  six 
tests  are  conducted,  savings  will  be  made  with  the  pay-full-time  option. 

Using  testers  from  the  PMWs  to  conduct  the  E2E  testing  would  possibly  lead  to 
the  time  loss  problem  faced  by  the  DEP  program,  as  the  PMW  testers  would  leave  after 
completion  of  the  testing  and  would  also  take  with  them  their  knowledge;  a  lack  of 
accountability  as  different  programs  from  which  testers  come  would  claim  effectiveness 
when,  in  actuality,  history  has  shown  otherwise;  finger  pointing  in  the  case  of  failure; 
difficulty  in  timely  coordination  and  flexibility  of  schedule  in  order  to  support  the  E2E 
testing  program;  and  possibly  not  meeting  the  schedule.  Rather  being  POM  funded,  the 
E2E  testing  program  secures  its  funding  from  taxing  the  PMWs.  As  such  funding  is 
uncertain,  the  E2E  testing  program  might  not  be  sustainable  in  the  future.  Eunding 
uncertainty  and  the  lack  of  a  complete  plan  to  establish  the  E2E  lab  contribute  to  the 
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delay  in  building  and  getting  the  E2E  lab  ready  for  testing.  Einally,  goal  programming 
ean  be  used  as  a  rigorous  approaeh  to  determining  the  goals  of  the  E2E  lab  in  a  timely 
fashion  (henee,  saving  money)  that  ean  meet  budgetary  eonstraints.  Eor  the  seenario 
treated  in  this  researeh,  the  planned  eost  for  building  the  E2E  lab,  whieh  is  $650,000, 
exeeeds  the  optimal  eost  obtained  with  the  goal  programming  approaeh,  whieh  is 
$647,000,  by  $3,000.  Whereas  money  savings  are  realized,  the  targeted  number  of  desks 
and  ehairs  is  short  by  one.  This  negligible  shortage  is  aeeeptable.  The  E2E  testing 
program  ean  use  this  approaeh  to  justify  the  funding  for  the  eurrent  year  and  future 
funding. 

It  is  reeommended  that  (1)  the  DEP  sites  be  funded  on  a  pay-per-test  basis  if  no 
more  than  six  tests  are  performed  in  a  year  and  on  a  full-time  basis  otherwise;  (2)  the  eore 
engineers  be  retained  to  mn  DEP  tests;  (3)  optimization  teehniques  in  general  and  goal 
programming  in  partieular  be  employed  in  planning  for  and,  speeifieally,  in  formulating  a 
plan  for  building  the  E2E  lab;  and  (4)  rigorous  approaehes — analytieal  and/or 
optimization  teehniques — be  employed  from  the  beginning  of  testing  programs  to  plan 
for  and  implement  the  testing  programs. 

Some  areas  for  further  researeh  inelude  (1)  extending  the  analysis  and 
optimization  teehniques  diseussed  in  this  thesis  to  determine  the  optimal  methods  of 
eondueting  tests  with  eonstraints  on  resourees  sueh  as  laboratories,  personnel,  sehedules, 
ete.  and  (2)  using  the  approaeh  espoused  in  this  thesis  as  a  foundation  for  additional 
researeh  and  studies  of  the  joint  distributed  testing/eoalition  distributed  testing. 
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APPENDIX  A.  DEP  TEST  DELAY  DATA  (FY02-FY06) 


FY02 

Test  1 

Start 

Loading 

(h:mm) 

End 

Loading 
(Start  to 
test) 
(h:mm) 

Different 

(h:mm) 

Week  1 

Day  1 

21:00 

2:22 

5:22 

Day  2 

21:00 

0:17 

S:17 

Day  S 

21:00 

22:12 

1:12 

Day  4 

21:00 

1:25 

4:25 

Day  5 

21:00 

22:28 

1:28 

Week  2 

Day  1 

21:00 

1:20 

4:20 

Day  2 

21:00 

2S:15 

2:15 

Day  S 

21:00 

4:20 

7:20 

Day  4 

21:00 

21:15 

0:15 

Day  5 

21:00 

22:10 

1:10 

Average 

3:06 

Test  2 

Week  1 

Day  1 

21:00 

0:S0 

3:30 

Day  2 

21:00 

2S:20 

2:20 

Day  S 

21:00 

2S:02 

2:02 

Week  2 

Day  1 

21:00 

0:01 

3:01 

Day  2 

21:00 

0:02 

3:02 

Day  S 

21:00 

22:11 

1:11 

Weeks 

Day  1 

21:00 

2S:09 

2:09 

Day  2 

21:00 

22:SS 

1:33 

Day  S 

21:00 

22:40 

1:40 

Day  4 

21:00 

21:51 

0:51 

Average 

2:07 

Test  3 

Week  1 

Day  1 

21:00 

5:00 

8:00 

Day  2 

21:00 

2:01 

5:01 

Day  S 

21:00 

1:0S 

4:03 

Week  2 

Day  1 

21:00 

5:00 

8:00 

Day  2 

21:00 

22:05 

1:05 

Day  S 

21:00 

2S:12 

2:12 

Weeks 

Day  1 

21:00 

4:15 

7:15 

Day  2 

21:00 

22:02 

1:02 

Day  S 

21:00 

2S:11 

2:11 

Day  4 

21:00 

22:12 

1:12 

Average 

4:00 

Test  4 
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Week  1 

Day  1 

21:00 

1:07 

4:07 

Day  2 

21:00 

0:57 

3:57 

Day  S 

21:00 

22:15 

1:15 

Week  2 

Day  1 

21:00 

22:S1 

1:S1 

Day  2 

21:00 

22:16 

1:16 

Day  S 

21:00 

5:00 

8:00 

Weeks 

Day  1 

21:00 

2S:0S 

2:0S 

Day  2 

21:00 

0:0S 

S:0S 

Day  S 

21:00 

5:00 

8:00 

Day  4 

21:00 

1:0S 

4:0S 

Average 

3:43 

Tests 

Week  1 

Day  1 

21:00 

22:22 

1:22 

Day  2 

21:00 

0:01 

3:01 

Day  S 

21:00 

22:01 

1:01 

Day  4 

21:00 

2S:05 

2:05 

Week  2 

Day  1 

21:00 

0:02 

3:02 

Day  2 

21:00 

2S:01 

2:01 

Day  S 

21:00 

0:07 

3:07 

Weeks 

Day  1 

21:00 

0:10 

3:10 

Day  2 

21:00 

0:05 

3:05 

Day  S 

21:00 

0:00 

3:00 

Average 

2:29 

Test  6 

Week  1 

Day  1 

21:00 

2S:55 

2:55 

Day  2 

21:00 

2S:15 

2:15 

Day  S 

21:00 

2S:42 

2:42 

Day  4 

21:00 

0:55 

3:55 

Day  5 

21:00 

2S:50 

2:50 

Week  2 

Day  1 

21:00 

0:52 

3:52 

Day  2 

21:00 

0:57 

3:57 

Day  S 

21:00 

0:50 

3:50 

Day  4 

21:00 

23:55 

2:55 

Day  5 

21:00 

23:30 

2:30 

Average 

3:10 

FY03 

Test  1 

Start 

Loading 

End 

Loading 
(Start  to 
test) 

Different 

Week  1 

Day  1 

21:00 

0:05 

3:05 

Day  2 

21:00 

22:35 

1:35 
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Day  2 

21:00 

0:11 

S:11 

Day  S 

21:00 

22:01 

1:01 

Day  4 

21:00 

22:45 

1:45 

Week  2 

Day  1 

21:00 

1:02 

4:02 

Day  2 

21:00 

22:S1 

1:S1 

Day  S 

21:00 

0:47 

S:47 

Weeks 

Day  1 

21:00 

0:20 

S:20 

Day  2 

21:00 

0:48 

S:48 

Day  S 

21:00 

0:S8 

S:S8 

Average 

2:48 

Test  6 

Week  1 

Day  1 

21:00 

5:00 

8:00 

Day  2 

21:00 

2:11 

5:11 

Day  S 

21:00 

1:SS 

4:SS 

Weeks 

Day  1 

21:00 

4:10 

7:10 

Day  2 

21:00 

22:S5 

1:S5 

Day  S 

21:00 

2S:12 

2:12 

Weeks 

Day  1 

21:00 

4:00 

7:00 

Day  2 

21:00 

22:42 

1:42 

Day  S 

21:00 

2S:S2 

2:S2 

Day  4 

21:00 

22:12 

1:12 

Average 

4:06 

Test  7 

Week  1 

Day  1 

21:00 

4:00 

7:00 

Day  2 

21:00 

2:11 

5:11 

Day  S 

21:00 

1:SS 

4:SS 

Day  4 

21:00 

4:00 

7:00 

Day  5 

21:00 

22:S5 

1:S5 

Weeks 

Day  1 

21:00 

2S:12 

2:12 

Day  2 

21:00 

2:10 

5:10 

Day  S 

21:00 

22:42 

1:42 

Day  4 

21:00 

2S:S2 

2:S2 

Day  5 

21:00 

22:12 

1:12 

Average 

3:48 

FY04 

Test  1 

Start 

Loading 

End 

Loading 
(Start  to 
test) 

Different 

Week  1 

Day  1 

21:00 

2:46 

5:46 

Day  2 

21:00 

2S:10 

2:10 

Day  S 

21:00 

5:00 

8:00 
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Week  2 


Average 


Week  1 


Week  2 


Average 


Week  1 


Week  2 


Weeks 


Average 


Week  1 


Week  2 


Weeks 


Average 


Week  1 


4 

2 

5 

2 

1 

2 

2 

2 

S 

2 

4 

2 

5 

2 

1 

2 

2 

2 

S 

2 

4 

2 

5 

2 

1 

2 

2 

2 

S 

2 

4 

2 

5 

2 

1 

2 

2 

2 

S 

2 

4 

2 

1 

2 

2 

2 

S 

2 

1 

2 

2 

2 

S 

2 

1 

2 

2 

2 

S 

2 

1 

2 

2 

2 

S 

2 

1 

2 

2 

2 

S 

2 

4 

2 

1 

2 

2 

2 

2S:55 

2S:10 

2:44 

1:19 

0:15 

0:20 

0:24 


22:02 

22:42 

2S:12 

0:01 

0:02 

22:11 

2S:09 

22:SS 

0:10 

0:0S 


1:07 

1:01 

0:27 

22:15 

22:51 

2S:16 

5:00 

22:0S 

5:00 

5:00 


1:07 

0:47 

22:15 

22:51 

2S:16 

5:00 

2S:0S 

0:2S 

5:00 

0:2S 


0:15 

2S:20 


Day  3 

21:00 

23:20 

2:20 

Week  2 

Day  1 

21:00 

0:51 

3:51 

Day  2 

21:00 

0:55 

3:55 

Day  3 

21:00 

22:51 

1:51 

Weeks 

Day  1 

21:00 

23:19 

2:19 

Day  2 

21:00 

22:33 

1:33 

Day  3 

21:00 

23:50 

2:50 

Day  4 

21:00 

21:51 

0:51 

Average 

2:30 

Test  6 

Week  1 

Day  1 

21:00 

1:07 

4:07 

Day  2 

21:00 

0:57 

3:57 

Day  3 

21:00 

22:15 

1:15 

Day  4 

21:00 

22:01 

1:01 

Day  5 

21:00 

23:10 

2:10 

Week  2 

Day  1 

21:00 

0:15 

3:15 

Day  2 

21:00 

23:03 

2:03 

Day  3 

21:00 

0:13 

3:13 

Day  4 

21:00 

1:00 

4:00 

Day  5 

21:00 

1:05 

4:05 

Average 

2:54 

FY05 

Test  1 

Start 

Loading 

End 

Loading 
(Start  to 
test) 

Different 

Week  1 

Day  1 

21:00 

3:55 

6:55 

Day  2 

21:00 

23:25 

2:25 

Day  3 

21:00 

23:04 

2:04 

Day  4 

21:00 

23:55 

2:55 

Week  2 

Day  1 

21:00 

2:54 

5:54 

Day  2 

21:00 

1:49 

4:49 

Day  3 

21:00 

23:57 

2:57 

Weeks 

Day  1 

21:00 

23:43 

2:43 

Day  2 

21:00 

23:35 

2:35 

Day  3 

21:00 

23:45 

2:45 

Average 

3:36 

Test  2 

Week  1 

Day  1 

21:00 

23:28 

2:28 

Day  2 

21:00 

22:42 

1:42 

Day  3 

21:00 

22:31 

1:31 

Day  4 

21:00 

0:11 

3:11 

Day  5 

21:00 

0:02 

3:02 

Week  2 

Day  1 

21:00 

0:08 

3:08 
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Day  2 

21:00 

23:30 

2:30 

Day  3 

21:00 

5:00 

8:00 

Day  4 

21:00 

0:15 

3:15 

Day  5 

21:00 

0:10 

3:10 

Average 

2:30 

Test  3 

Week  1 

Day  1 

21:00 

1:30 

4:30 

Day  2 

21:00 

0:40 

3:40 

Day  3 

21:00 

5:00 

8:00 

Day  4 

21:00 

23:31 

2:31 

Day  5 

21:00 

1:47 

4:47 

Week  2 

Day  1 

21:00 

23:37 

2:37 

Day  2 

21:00 

0:23 

3:23 

Day  3 

21:00 

23:38 

2:38 

Day  4 

21:00 

0:45 

3:45 

Day  5 

21:00 

23:39 

2:39 

Average 

4:00 

FY06 

Test  1 

Start 

Loading 

End 

Loading 
(Start  to 
test) 

Different 

Week  1 

Day  1 

21:00 

0:04 

3:04 

Day  2 

21:00 

22:00 

1:00 

Day  3 

21:00 

22:09 

1:09 

Day  4 

21:00 

0:13 

3:13 

Day  5 

21:00 

0:03 

3:03 

Week  2 

Day  1 

21:00 

22:32 

1:32 

Day  2 

21:00 

0:15 

3:15 

Day  3 

21:00 

21:59 

0:59 

Day  4 

21:00 

0:03 

3:03 

Day  5 

21:00 

0:13 

3:13 

Average 

2:12 

Test  2 

Week  1 

Day  1 

21:00 

0:15 

3:15 

Day  2 

21:00 

22:25 

1:25 

Day  3 

21:00 

22:04 

1:04 

Day  4 

21:00 

22:49 

1:49 

Week  2 

Day  1 

21:00 

0:34 

3:34 

Day  2 

21:00 

0:04 

3:04 

Day  3 

21:00 

23:17 

2:17 

Week  3 

Day  1 

21:00 

22:53 

1:53 

Day  2 

21:00 

22:31 

1:31 

Day  3 

21:00 

23:15 

2:15 
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